[MDEV-30084] Shutdown hangs in some tests Created: 2022-11-24  Updated: 2023-07-04  Resolved: 2023-07-04

Status: Closed
Project: MariaDB Server
Component/s: Server, Tests
Affects Version/s: 10.5, 10.6, 10.7, 10.8, 10.9, 10.10, 10.11
Fix Version/s: 10.6.15, 10.9.8, 10.10.6, 10.11.5, 11.0.3

Type: Bug Priority: Critical
Reporter: Marko Mäkelä Assignee: Sergei Golubchik
Resolution: Fixed Votes: 0
Labels: hang

Issue Links:
Problem/Incident
is caused by MDEV-29368 Assertion `trx->mysql_thd == thd' fai... Closed
Relates
relates to MDEV-28931 MTR prints detailed stack trace uncon... Closed
relates to MDEV-30242 MTR fails to report stack traces of a... Closed

 Description   

The test main.bootstrap_innodb is often failing with a message like this:

10.7 22ab79c43079d9fe0ca136148fa6fd05a5aa4d5a

main.bootstrap_innodb 'innodb'           w23 [ fail ]  Found warnings/errors in server log file!
        Test ended at 2022-11-24 06:57:37
line
Attempting backtrace. You can use the following information to find out

In the corresponding server error log, we can find a hint of the real cause:

CURRENT_TEST: main.bootstrap_innodb
2022-11-24  6:56:34 0 [Note] /home/buildbot/amd64-ubuntu-2004-debug/build/sql/mariadbd (initiated by: root[root] @ localhost []): Normal shutdown
221124  6:57:34 [ERROR] mysqld got signal 6 ;

Because 60 seconds (1 minute) passed between the two messages, the server process must have been killed by mysqlclient. That is, the 60-second shutdown timeout elapsed.

In either log, I am only seeing the stack trace of a single thread. The cause of the hang is thus unknown.

Another hang:

10.9 d48db97d0abec7b310fed362da93c575cbbd7124

main.statistics_upgrade_not_done 'innodb' w27 [ fail ]  Found warnings/errors in server log file!
        Test ended at 2022-11-24 07:07:19
line
Attempting backtrace. You can use the following information to find out

Again, the server error log shows a time difference of 60 seconds (1 minute) between shutdown initiation and the kill -ABRT by the test harness:

CURRENT_TEST: main.statistics_upgrade_not_done
2022-11-24  7:06:16 317 [ERROR] Incorrect definition of table mysql.column_stats: expected column 'hist_type' at position 9 to have type enum('SINGLE_PREC_HB','DOUBLE_PREC_HB','JSON_HB'), found type enum('SINGLE_PREC_HB','DOUBLE_PREC_HB').
2022-11-24  7:06:16 317 [ERROR] Incorrect definition of table mysql.column_stats: expected column 'histogram' at position 10 to have type longblob, found type varbinary(255).
2022-11-24  7:06:16 317 [ERROR] Incorrect definition of table mysql.column_stats: expected column 'hist_type' at position 9 to have type enum('SINGLE_PREC_HB','DOUBLE_PREC_HB','JSON_HB'), found type enum('SINGLE_PREC_HB','DOUBLE_PREC_HB').
2022-11-24  7:06:16 317 [ERROR] Incorrect definition of table mysql.column_stats: expected column 'histogram' at position 10 to have type longblob, found type varbinary(255).
2022-11-24  7:06:16 0 [Note] /home/buildbot/amd64-ubuntu-2204-debug-ps/build/sql/mariadbd (initiated by: root[root] @ localhost []): Normal shutdown
221124  7:07:16 [ERROR] mysqld got signal 6 ;

Because both hangs occurred on mandatory staging builders and required the builds to be rerun (incurring an extra wait of over 30 minutes) I think that this needs to be treated at a high priority.



 Comments   
Comment by Marko Mäkelä [ 2022-12-15 ]

Recent occurrence:

st-10.10-merge fc9d350ab7beff384926d9975dee9ab52b4cfe94

main.statistics_upgrade_not_done 'innodb' w1 [ fail ]  Found warnings/errors in server log file!
        Test ended at 2022-12-15 06:42:39
line
Attempting backtrace. You can use the following information to find out
^ Found warnings in /dev/shm/var/1/log/mysqld.1.err
ok
 
 - found 'core.20820' (0/0)
Core generated by '/usr/sbin/mariadbd'
Output from gdb follows. The first stack trace is from the failing thread.
The following stack traces are from all threads (so the failing one is
duplicated).
--------------------------
[New LWP 20820]
[New LWP 20823]
[New LWP 20824]
[New LWP 20835]
[New LWP 21223]
[New LWP 20827]
[New LWP 20821]
[New LWP 20822]
[New LWP 20838]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/sbin/mariadbd --defaults-group-suffix=.1 --defaults-file=/dev/shm/var/1/my'.
Program terminated with signal 6, Aborted.
#0  0x00007f46cebf69b1 in pthread_kill () from /lib64/libpthread.so.0
 
Thread 9 (Thread 0x7f46d0460700 (LWP 20838)):
#0  0x00007f46ce04f3fa in sigwaitinfo () from /lib64/libc.so.6
#1  0x0000560333bdf75f in my_sigwait (code=<synthetic pointer>, sig=0x7f46d045f8f4, set=0x7f46d045fa10) at /usr/src/debug/MariaDB-/src_0/include/my_pthread.h:195
#2  signal_hand (arg=arg@entry=0x0) at /usr/src/debug/MariaDB-/src_0/sql/mysqld.cc:3201
#3  0x000056033422e63e in pfs_spawn_thread (arg=0x56033b8f8b08) at /usr/src/debug/MariaDB-/src_0/storage/perfschema/pfs.cc:2201
#4  0x00007f46cebf1e25 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f46ce11134d in clone () from /lib64/libc.so.6
 
Thread 8 (Thread 0x7f46cab2d700 (LWP 20822)):
#0  0x00007f46cebf5cf2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00005603346c03d0 in safe_cond_timedwait (cond=cond@entry=0x560335ced360 <COND_checkpoint>, mp=mp@entry=0x560335ced3a0 <LOCK_checkpoint>, abstime=abstime@entry=0x7f46cab2caa0, file=file@entry=0x560334b3bd58 "/home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.10.3/mysys/my_thr_init.c", line=line@entry=609) at /usr/src/debug/MariaDB-/src_0/mysys/thr_mutex.c:548
#2  0x00005603346ba8c4 in psi_cond_timedwait (that=0x560335ced360 <COND_checkpoint>, mutex=0x560335ced3a0 <LOCK_checkpoint>, abstime=abstime@entry=0x7f46cab2caa0, file=file@entry=0x5603349de420 "/home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.10.3/storage/maria/ma_servicethread.c", line=line@entry=116) at /usr/src/debug/MariaDB-/src_0/mysys/my_thr_init.c:609
#3  0x00005603341b46ea in inline_mysql_cond_timedwait (src_line=116, src_file=0x5603349de420 "/home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.10.3/storage/maria/ma_servicethread.c", abstime=0x7f46cab2caa0, mutex=<optimized out>, that=<optimized out>) at /usr/src/debug/MariaDB-/src_0/include/mysql/psi/mysql_thread.h:1086
#4  my_service_thread_sleep (control=control@entry=0x5603353ba780 <checkpoint_control>, sleep_time=sleep_time@entry=30000000000) at /usr/src/debug/MariaDB-/src_0/storage/maria/ma_servicethread.c:115
#5  0x00005603341ac2a0 in ma_checkpoint_background (arg=arg@entry=0x1e) at /usr/src/debug/MariaDB-/src_0/storage/maria/ma_checkpoint.c:725
#6  0x000056033422e63e in pfs_spawn_thread (arg=0x56033b566788) at /usr/src/debug/MariaDB-/src_0/storage/perfschema/pfs.cc:2201
#7  0x00007f46cebf1e25 in start_thread () from /lib64/libpthread.so.0
#8  0x00007f46ce11134d in clone () from /lib64/libc.so.6
 
Thread 7 (Thread 0x7f46d0594700 (LWP 20821)):
#0  0x00007f46cebf5cf2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00005603346c03d0 in safe_cond_timedwait (cond=cond@entry=0x560335d80560 <COND_timer>, mp=mp@entry=0x560335d805a0 <LOCK_timer>, abstime=abstime@entry=0x7f46d0593b10, file=file@entry=0x560334b3bd58 "/home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.10.3/mysys/my_thr_init.c", line=line@entry=609) at /usr/src/debug/MariaDB-/src_0/mysys/thr_mutex.c:548
#2  0x00005603346ba8c4 in psi_cond_timedwait (that=that@entry=0x560335d80560 <COND_timer>, mutex=mutex@entry=0x560335d805a0 <LOCK_timer>, abstime=abstime@entry=0x7f46d0593b10, file=file@entry=0x560334b3ce50 "/home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.10.3/mysys/thr_timer.c", line=line@entry=321) at /usr/src/debug/MariaDB-/src_0/mysys/my_thr_init.c:609
#3  0x00005603346c0a5a in inline_mysql_cond_timedwait (src_line=321, src_file=0x560334b3ce50 "/home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.10.3/mysys/thr_timer.c", abstime=0x7f46d0593b10, mutex=0x560335d805a0 <LOCK_timer>, that=0x560335d80560 <COND_timer>) at /usr/src/debug/MariaDB-/src_0/include/mysql/psi/mysql_thread.h:1086
#4  timer_handler (arg=arg@entry=0x0) at /usr/src/debug/MariaDB-/src_0/mysys/thr_timer.c:321
#5  0x000056033422e63e in pfs_spawn_thread (arg=0x56033b526088) at /usr/src/debug/MariaDB-/src_0/storage/perfschema/pfs.cc:2201
#6  0x00007f46cebf1e25 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f46ce11134d in clone () from /lib64/libc.so.6
 
Thread 6 (Thread 0x7f46c812a700 (LWP 20827)):
#0  0x00007f46cebf5cf2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00005603346498cf in __gthread_cond_timedwait (__abs_timeout=0x7f46c8129a90, __mutex=<optimized out>, __cond=0x56033b99a000) at /usr/include/c++/4.8.2/x86_64-redhat-linux/bits/gthr-default.h:871
#2  __wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=..., __lock=..., this=0x56033b99a000) at /usr/include/c++/4.8.2/condition_variable:160
#3  wait_until<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=..., __lock=..., this=0x56033b99a000) at /usr/include/c++/4.8.2/condition_variable:100
#4  wait_for<long, std::ratio<1l, 1000l> > (__rtime=..., __lock=..., this=0x56033b99a000) at /usr/include/c++/4.8.2/condition_variable:132
#5  tpool::thread_pool_generic::wait_for_tasks (this=this@entry=0x56033b566a80, lk=..., thread_data=thread_data@entry=0x56033b99a000) at /usr/src/debug/MariaDB-/src_0/tpool/tpool_generic.cc:480
#6  0x000056033464a2dd in tpool::thread_pool_generic::get_task (this=this@entry=0x56033b566a80, thread_var=thread_var@entry=0x56033b99a000, t=t@entry=0x7f46c8129b38) at /usr/src/debug/MariaDB-/src_0/tpool/tpool_generic.cc:533
#7  0x000056033464a41e in tpool::thread_pool_generic::worker_main (this=0x56033b566a80, thread_var=0x56033b99a000) at /usr/src/debug/MariaDB-/src_0/tpool/tpool_generic.cc:578
#8  0x000056033464be6d in operator()<tpool::worker_data*, void> (__object=<optimized out>, this=<optimized out>) at /usr/include/c++/4.8.2/functional:601
#9  _M_invoke<0ul, 1ul> (this=<optimized out>) at /usr/include/c++/4.8.2/functional:1732
#10 operator() (this=<optimized out>) at /usr/include/c++/4.8.2/functional:1720
#11 std::thread::_Impl<std::_Bind_simple<std::_Mem_fn<void (tpool::thread_pool_generic::*)(tpool::worker_data*)> (tpool::thread_pool_generic*, tpool::worker_data*)> >::_M_run() (this=<optimized out>) at /usr/include/c++/4.8.2/thread:115
#12 0x00007f46ce7932b0 in ?? () from /lib64/libstdc++.so.6
#13 0x00007f46cebf1e25 in start_thread () from /lib64/libpthread.so.0
#14 0x00007f46ce11134d in clone () from /lib64/libc.so.6
 
Thread 5 (Thread 0x7f46c388d700 (LWP 21223)):
#0  0x00007f46cebf5cf2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00005603346c03d0 in safe_cond_timedwait (cond=cond@entry=0x5603354b9320 <debug_sync_global+32>, mp=mp@entry=0x5603354b9358 <debug_sync_global+88>, abstime=abstime@entry=0x7f46c388c9b0, file=file@entry=0x560334b3bd58 "/home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.10.3/mysys/my_thr_init.c", line=line@entry=609) at /usr/src/debug/MariaDB-/src_0/mysys/thr_mutex.c:548
#2  0x00005603346ba8c4 in psi_cond_timedwait (that=0x5603354b9320 <debug_sync_global+32>, mutex=0x5603354b9358 <debug_sync_global+88>, abstime=0x7f46c388c9b0, file=<optimized out>, line=<optimized out>) at /usr/src/debug/MariaDB-/src_0/mysys/my_thr_init.c:609
#3  0x0000560333d3e193 in inline_mysql_cond_timedwait (src_line=1482, src_file=0x5603347d9a88 "/home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.10.3/sql/debug_sync.cc", abstime=0x7f46c388c9b0, mutex=0x5603354b9358 <debug_sync_global+88>, that=0x5603354b9320 <debug_sync_global+32>) at /usr/src/debug/MariaDB-/src_0/include/mysql/psi/mysql_thread.h:1086
#4  debug_sync_execute (action=0x56033d59e088, thd=0x56033de6a088) at /usr/src/debug/MariaDB-/src_0/sql/debug_sync.cc:1482
#5  debug_sync (thd=0x56033de6a088, sync_point_name=<optimized out>, name_len=<optimized out>) at /usr/src/debug/MariaDB-/src_0/sql/debug_sync.cc:1588
#6  0x00005603342c6408 in innobase_close_connection (hton=<optimized out>, thd=0x56033de6a088) at /usr/src/debug/MariaDB-/src_0/storage/innobase/handler/ha_innodb.cc:4934
#7  0x0000560333fc5e29 in ha_close_connection (thd=thd@entry=0x56033de6a088) at /usr/src/debug/MariaDB-/src_0/sql/handler.cc:907
#8  0x0000560333ca2c02 in THD::free_connection (this=this@entry=0x56033de6a088) at /usr/src/debug/MariaDB-/src_0/sql/sql_class.cc:1602
#9  0x0000560333be1d66 in unlink_thd (thd=0x56033de6a088) at /usr/src/debug/MariaDB-/src_0/sql/mysqld.cc:2778
#10 0x0000560333e568bd in do_handle_one_connection (connect=<optimized out>, connect@entry=0x56033778ecc8, put_in_cache=put_in_cache@entry=true) at /usr/src/debug/MariaDB-/src_0/sql/sql_connect.cc:1427
#11 0x0000560333e56ae7 in handle_one_connection (arg=arg@entry=0x56033778ecc8) at /usr/src/debug/MariaDB-/src_0/sql/sql_connect.cc:1318
#12 0x000056033422e63e in pfs_spawn_thread (arg=0x56033b8f8b08) at /usr/src/debug/MariaDB-/src_0/storage/perfschema/pfs.cc:2201
#13 0x00007f46cebf1e25 in start_thread () from /lib64/libpthread.so.0
#14 0x00007f46ce11134d in clone () from /lib64/libc.so.6
 
Thread 4 (Thread 0x7f46d04aa700 (LWP 20835)):
#0  0x00007f46cebf5945 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00005603346c01de in safe_cond_wait (cond=cond@entry=0x5603354b7d40 <COND_manager>, mp=mp@entry=0x5603354b7d80 <LOCK_manager>, file=file@entry=0x560334b3bd58 "/home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.10.3/mysys/my_thr_init.c", line=line@entry=596) at /usr/src/debug/MariaDB-/src_0/mysys/thr_mutex.c:494
#2  0x00005603346ba83b in psi_cond_wait (that=0x5603354b7d40 <COND_manager>, mutex=0x5603354b7d80 <LOCK_manager>, file=<optimized out>, line=<optimized out>) at /usr/src/debug/MariaDB-/src_0/mysys/my_thr_init.c:596
#3  0x0000560333cfabfe in inline_mysql_cond_wait (src_line=103, src_file=0x5603347cd2a0 "/home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.10.3/sql/sql_manager.cc", mutex=0x5603354b7d80 <LOCK_manager>, that=0x5603354b7d40 <COND_manager>) at /usr/src/debug/MariaDB-/src_0/include/mysql/psi/mysql_thread.h:1070
#4  handle_manager (arg=arg@entry=0x0) at /usr/src/debug/MariaDB-/src_0/sql/sql_manager.cc:103
#5  0x000056033422e63e in pfs_spawn_thread (arg=0x56033b8f9208) at /usr/src/debug/MariaDB-/src_0/storage/perfschema/pfs.cc:2201
#6  0x00007f46cebf1e25 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f46ce11134d in clone () from /lib64/libc.so.6
 
Thread 3 (Thread 0x7f46c932b700 (LWP 20824)):
#0  0x00007f46cebf5945 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00005603346c01de in safe_cond_wait (cond=0x5603353d8788 <buf_pool+17544>, mp=0x5603353d8680 <buf_pool+17280>, file=0x560334af2330 "/home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.10.3/storage/innobase/buf/buf0flu.cc", line=2339) at /usr/src/debug/MariaDB-/src_0/mysys/thr_mutex.c:494
#2  0x000056033455e689 in buf_flush_page_cleaner () at /usr/src/debug/MariaDB-/src_0/storage/innobase/buf/buf0flu.cc:2339
#3  0x000056033455ebc1 in _M_invoke<> (this=<optimized out>) at /usr/include/c++/4.8.2/functional:1732
#4  operator() (this=<optimized out>) at /usr/include/c++/4.8.2/functional:1720
#5  std::thread::_Impl<std::_Bind_simple<void (*())()> >::_M_run() (this=<optimized out>) at /usr/include/c++/4.8.2/thread:115
#6  0x00007f46ce7932b0 in ?? () from /lib64/libstdc++.so.6
#7  0x00007f46cebf1e25 in start_thread () from /lib64/libpthread.so.0
#8  0x00007f46ce11134d in clone () from /lib64/libc.so.6
 
Thread 2 (Thread 0x7f46ca32c700 (LWP 20823)):
#0  0x00007f46ce10b7f9 in syscall () from /lib64/libc.so.6
#1  0x000056033464d7b5 in my_getevents (ev=0x7f46ca329b30, nr=256, min_nr=1, ctx=<optimized out>) at /usr/src/debug/MariaDB-/src_0/tpool/aio_linux.cc:63
#2  tpool::aio_linux::getevent_thread_routine (aio=0x5603370e61a0) at /usr/src/debug/MariaDB-/src_0/tpool/aio_linux.cc:104
#3  0x000056033464d5b4 in _M_invoke<0ul> (this=<optimized out>) at /usr/include/c++/4.8.2/functional:1732
#4  operator() (this=<optimized out>) at /usr/include/c++/4.8.2/functional:1720
#5  std::thread::_Impl<std::_Bind_simple<void (*(tpool::aio_linux*))(tpool::aio_linux*)> >::_M_run() (this=<optimized out>) at /usr/include/c++/4.8.2/thread:115
#6  0x00007f46ce7932b0 in ?? () from /lib64/libstdc++.so.6
#7  0x00007f46cebf1e25 in start_thread () from /lib64/libpthread.so.0
#8  0x00007f46ce11134d in clone () from /lib64/libc.so.6
 
Thread 1 (Thread 0x7f46d0597ac0 (LWP 20820)):
#0  0x00007f46cebf69b1 in pthread_kill () from /lib64/libpthread.so.0
#1  0x00005603346bb854 in my_write_core (sig=6) at /usr/src/debug/MariaDB-/src_0/mysys/stacktrace.c:424
#2  0x0000560333fc154d in handle_fatal_signal (sig=6) at /usr/src/debug/MariaDB-/src_0/sql/signal_handler.cc:355
#3  <signal handler called>
#4  0x00007f46ce1087a3 in select () from /lib64/libc.so.6
#5  0x00005603346b8974 in my_sleep (m_seconds=<optimized out>) at /usr/src/debug/MariaDB-/src_0/mysys/my_sleep.c:29
#6  0x0000560333bebe33 in close_connections () at /usr/src/debug/MariaDB-/src_0/sql/mysqld.cc:1835
#7  mysqld_main (argc=156, argv=0x56033711aee0) at /usr/src/debug/MariaDB-/src_0/sql/mysqld.cc:5960
#8  0x0000560333bdd03e in main (argc=<optimized out>, argv=<optimized out>) at /usr/src/debug/MariaDB-/src_0/sql/main.cc:34

The only interesting InnoDB thread is waiting in

    DEBUG_SYNC(thd, "innobase_connection_closed");

That synchronization point is being used in the test main.kill_debug. In this execution, that test was indeed executed on the same worker w1, without any server restart in between.

The test main.kill_debug was added rather recently in MDEV-29368 by serg.

For this failure of main.bootstrap_innodb, we also had the test main.kill_debug running on the same worker, without a server restart in between.

Comment by Marko Mäkelä [ 2022-12-16 ]

The loss of necessary diagnostic information was caused by MDEV-28931 and addressed by MDEV-30242.

Comment by Daniel Black [ 2023-01-24 ]

build.

var.tar.gz (from mtr, with core, no exe however)

main.statistics_upgrade_not_done - 10.10-61c95da67f5dec6e1e8fe80146dc34b7074f0a22

main.statistics_upgrade_not_done 'innodb' w20 [ fail ]  Found warnings/errors in server log file!
        Test ended at 2023-01-23 23:09:24
line
Attempting backtrace. You can use the following information to find out
^ Found warnings in /home/buildbot/amd64-ubuntu-2004-debug/build/mysql-test/var/20/log/mysqld.1.err
ok
 - found 'core' (0/1)
Core generated by '/home/buildbot/amd64-ubuntu-2004-debug/build/sql/mariadbd'
Output from gdb follows. The first stack trace is from the failing thread.
The following stack traces are from all threads (so the failing one is
duplicated).
--------------------------
[New LWP 307712]
[New LWP 307784]
[New LWP 307790]
[New LWP 307859]
[New LWP 307826]
[New LWP 314484]
[New LWP 338968]
[New LWP 338971]
[New LWP 307860]
[New LWP 311279]
[New LWP 307827]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/home/buildbot/amd64-ubuntu-2004-debug/build/sql/mariadbd --defaults-group-suff'.
Program terminated with signal SIGABRT, Aborted.
#0  0x00007f4b0c951f68 in pthread_kill () from /lib/x86_64-linux-gnu/libpthread.so.0
[Current thread is 1 (Thread 0x7f4b0c25f940 (LWP 307712))]
#0  0x00007f4b0c951f68 in pthread_kill () from /lib/x86_64-linux-gnu/libpthread.so.0
#1  0x000056390d68b39e in my_write_core (sig=6) at /home/buildbot/amd64-ubuntu-2004-debug/build/mysys/stacktrace.c:424
#2  0x000056390ce2871f in handle_fatal_signal (sig=6) at /home/buildbot/amd64-ubuntu-2004-debug/build/sql/signal_handler.cc:357
#3  <signal handler called>
#4  0x00007f4b0c50ffcb in select () from /lib/x86_64-linux-gnu/libc.so.6
#5  0x000056390d687816 in my_sleep (m_seconds=1000) at /home/buildbot/amd64-ubuntu-2004-debug/build/mysys/my_sleep.c:29
#6  0x000056390c904e87 in close_connections () at /home/buildbot/amd64-ubuntu-2004-debug/build/sql/mysqld.cc:1837
#7  0x000056390c90ce8d in mysqld_main (argc=156, argv=0x56390ee16dd0) at /home/buildbot/amd64-ubuntu-2004-debug/build/sql/mysqld.cc:5955
#8  0x000056390c9019ed in main (argc=23, argv=0x7ffe54d0e2e8) at /home/buildbot/amd64-ubuntu-2004-debug/build/sql/main.cc:34
Thread 11 (Thread 0x7f4b058b3700 (LWP 307827)):
#0  0x00007f4b0c950376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#1  0x000056390d690379 in safe_cond_wait (cond=0x56390e255888 <buf_pool+17544>, mp=0x56390e255780 <buf_pool+17280>, file=0x56390db95d10 "/home/buildbot/amd64-ubuntu-2004-debug/build/storage/innobase/buf/buf0flu.cc", line=2339) at /home/buildbot/amd64-ubuntu-2004-debug/build/mysys/thr_mutex.c:494
        error = 32587
        save_state = {global = {__data = {__lock = 1, __count = 0, __owner = 307827, __nusers = 1, __kind = 2, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = "\001\000\000\000\000\000\000\000s\262\004\000\001\000\000\000\002", '\000' <repeats 22 times>, __align = 1}, mutex = {__data = {__lock = 1, __count = 0, __owner = 307827, __nusers = 1, __kind = 3, __spins = 22, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = "\001\000\000\000\000\000\000\000s\262\004\000\001\000\000\000\003\000\000\000\026", '\000' <repeats 18 times>, __align = 1}, file = 0x56390db95d10 "/home/buildbot/amd64-ubuntu-2004-debug/build/storage/innobase/buf/buf0flu.cc", name = 0x56390db8d473 "flush_list_mutex", line = 2381, count = 0, create_flags = 0, active_flags = 0, id = 20, thread = 139960192284416, locked_mutex = 0x56390f187e48, used_mutex = 0x56390f187ec8, prev = 0x0, next = 0x0}
#2  0x000056390d4fa41d in buf_flush_page_cleaner () at /home/buildbot/amd64-ubuntu-2004-debug/build/storage/innobase/buf/buf0flu.cc:2339
        soft_lsn_limit = 0
        oldest_lsn = 22193749
        dirty_blocks = 91
        dirty_pct = 18.055555555555557
        idle_flush = false
        n_flushed = 161
        __PRETTY_FUNCTION__ = "void buf_flush_page_cleaner()"
        last_pages = 161
        abstime = {tv_sec = 1674515293, tv_nsec = 824961000}
        lsn_limit = 0
        last_activity_count = 0
#3  0x000056390d4ffc2b in std::__invoke_impl<void, void (*)()> (__f=@0x56390ee2cee8: 0x56390d4fa0cc <buf_flush_page_cleaner()>) at /usr/include/c++/9/bits/invoke.h:60
No locals.
#4  0x000056390d4ffbd7 in std::__invoke<void (*)()> (__fn=@0x56390ee2cee8: 0x56390d4fa0cc <buf_flush_page_cleaner()>) at /usr/include/c++/9/bits/invoke.h:95
No locals.
#5  0x000056390d4ffb78 in std::thread::_Invoker<std::tuple<void (*)()> >::_M_invoke<0ul> (this=0x56390ee2cee8) at /usr/include/c++/9/thread:244
No locals.
#6  0x000056390d4ffb4a in std::thread::_Invoker<std::tuple<void (*)()> >::operator() (this=0x56390ee2cee8) at /usr/include/c++/9/thread:251
No locals.
#7  0x000056390d4ffb2a in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (*)()> > >::_M_run (this=0x56390ee2cee0) at /usr/include/c++/9/thread:195
No locals.
#8  0x00007f4b0c82fde4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#9  0x00007f4b0c949609 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#10 0x00007f4b0c51a133 in clone () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
Thread 10 (Thread 0x7f4a6effd700 (LWP 311279)):
#0  0x00007f4b0c9507d1 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#1  0x000056390d5f39e7 in __gthread_cond_timedwait (__cond=0x56390f07a9f0, __mutex=0x56390f0791d8, __abs_timeout=0x7f4a6effca50) at /usr/include/x86_64-linux-gnu/c++/9/bits/gthr-default.h:872
No locals.
#2  0x000056390d5f6615 in std::condition_variable::__wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x56390f07a9f0, __lock=@0x7f4a6effcb70: {_M_device = 0x56390f0791d8, _M_owns = true}, __atime=@0x7f4a6effcab0: {__d = {__r = 1674515419064606142}}) at /usr/include/c++/9/condition_variable:188
        __s = {__d = {__r = 1674515419}}
        __ns = {__r = 64606142}
        __ts = {tv_sec = 1674515419, tv_nsec = 64606142}
#3  0x000056390d5f5821 in std::condition_variable::wait_until<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x56390f07a9f0, __lock=@0x7f4a6effcb70: {_M_device = 0x56390f0791d8, _M_owns = true}, __atime=@0x7f4a6effcb08: {__d = {__r = 43489355440884650}}) at /usr/include/c++/9/condition_variable:121
        __c_entry = {__d = {__r = 43489295440884810}}
        __s_entry = {__d = {__r = 1674515359064606302}}
        __delta = {__r = 59999999840}
        __s_atime = {__d = {__r = 1674515419064606142}}
#4  0x000056390d5f4aaa in std::condition_variable::wait_for<long, std::ratio<1l, 1000l> > (this=0x56390f07a9f0, __lock=@0x7f4a6effcb70: {_M_device = 0x56390f0791d8, _M_owns = true}, __rtime=@0x56390f079200: {__r = 60000}) at /usr/include/c++/9/condition_variable:152
        __reltime = {__r = 60000000000}
#5  0x000056390d5f2179 in tpool::thread_pool_generic::wait_for_tasks (this=0x56390f0790c0, lk=@0x7f4a6effcb70: {_M_device = 0x56390f0791d8, _M_owns = true}, thread_data=0x56390f07a9f0) at /home/buildbot/amd64-ubuntu-2004-debug/build/tpool/tpool_generic.cc:480
        __PRETTY_FUNCTION__ = "bool tpool::thread_pool_generic::wait_for_tasks(std::unique_lock<std::mutex>&, tpool::worker_data*)"
#6  0x000056390d5f2389 in tpool::thread_pool_generic::get_task (this=0x56390f0790c0, thread_var=0x56390f07a9f0, t=0x7f4a6effcbb8) at /home/buildbot/amd64-ubuntu-2004-debug/build/tpool/tpool_generic.cc:533
        lk = {_M_device = 0x56390f0791d8, _M_owns = true}
        __PRETTY_FUNCTION__ = "bool tpool::thread_pool_generic::get_task(tpool::worker_data*, tpool::task**)"
#7  0x000056390d5f2623 in tpool::thread_pool_generic::worker_main (this=0x56390f0790c0, thread_var=0x56390f07a9f0) at /home/buildbot/amd64-ubuntu-2004-debug/build/tpool/tpool_generic.cc:578
        task = 0x56390f1db9e0
#8  0x000056390d5f8a36 in std::__invoke_impl<void, void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> (__f=@0x7f4ae8004d18: (void (tpool::thread_pool_generic::*)(tpool::thread_pool_generic * const, tpool::worker_data *)) 0x56390d5f25be <tpool::thread_pool_generic::worker_main(tpool::worker_data*)>, __t=@0x7f4ae8004d10: 0x56390f0790c0) at /usr/include/c++/9/bits/invoke.h:73
No locals.
#9  0x000056390d5f8918 in std::__invoke<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> (__fn=@0x7f4ae8004d18: (void (tpool::thread_pool_generic::*)(tpool::thread_pool_generic * const, tpool::worker_data *)) 0x56390d5f25be <tpool::thread_pool_generic::worker_main(tpool::worker_data*)>) at /usr/include/c++/9/bits/invoke.h:95
No locals.
#10 0x000056390d5f8837 in std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> >::_M_invoke<0ul, 1ul, 2ul> (this=0x7f4ae8004d08) at /usr/include/c++/9/thread:244
No locals.
#11 0x000056390d5f87d2 in std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> >::operator() (this=0x7f4ae8004d08) at /usr/include/c++/9/thread:251
No locals.
#12 0x000056390d5f87b2 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> > >::_M_run (this=0x7f4ae8004d00) at /usr/include/c++/9/thread:195
No locals.
#13 0x00007f4b0c82fde4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#14 0x00007f4b0c949609 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#15 0x00007f4b0c51a133 in clone () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
Thread 9 (Thread 0x7f4b0425d700 (LWP 307860)):
#0  0x00007f4b0c43f1a2 in sigtimedwait () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
#1  0x000056390c901ac2 in my_sigwait (set=0x7f4b0425cc20, sig=0x7f4b0425cc04, code=0x7f4b0425cc08) at /home/buildbot/amd64-ubuntu-2004-debug/build/include/my_pthread.h:195
        siginfo = {si_signo = 0, si_errno = 0, si_code = 0, __pad0 = 3195, _sifields = {_pad = {225994064, 22073, 239534720, 22073, 7, 25263279, 138540864, 0, 69585888, 32587, 0, 0, 69585888, 32587, 210771100, 22073, 69586368, 32587, 210772263, 3195, 225994064, 22073, 239534720, 22073, 225994064, 22073, -39689472, 1693908237}, _kill = {si_pid = 225994064, si_uid = 22073}, _timer = {si_tid = 225994064, si_overrun = 22073, si_sigval = {sival_int = 239534720, sival_ptr = 0x56390e470280 <LOCK_start_thread>}}, _rt = {si_pid = 225994064, si_uid = 22073, si_sigval = {sival_int = 239534720, sival_ptr = 0x56390e470280 <LOCK_start_thread>}}, _sigchld = {si_pid = 225994064, si_uid = 22073, si_status = 239534720, si_utime = 108504957094723591, si_stime = 138540864}, _sigfault = {si_addr = 0x56390d786550, si_addr_lsb = 640, _bounds = {_addr_bnd = {_lower = 0x1817caf00000007, _upper = 0x841f740}, _pkey = 7}}, _sigpoll = {si_band = 94803039118672, si_fd = 239534720}, _sigsys = {_call_addr = 0x56390d786550, _syscall = 239534720, _arch = 22073}}}
#2  0x000056390c9078b0 in signal_hand (arg=0x0) at /home/buildbot/amd64-ubuntu-2004-debug/build/sql/mysqld.cc:3203
        error = 22073
        origin = 258361632
        set = {__val = {548869, 0 <repeats 15 times>}}
        sig = 0
#3  0x000056390d1112bb in pfs_spawn_thread (arg=0x56390f664938) at /home/buildbot/amd64-ubuntu-2004-debug/build/storage/perfschema/pfs.cc:2201
        typed_arg = 0x56390f664938
        user_arg = 0x0
        user_start_routine = 0x56390c907729 <signal_hand(void*)>
        pfs = 0x7f4b0bba0f40
        klass = 0x56390ee32d40
#4  0x00007f4b0c949609 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#5  0x00007f4b0c51a133 in clone () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
Thread 8 (Thread 0x7f4a88ff9700 (LWP 338971)):
#0  0x00007f4b0c9507d1 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#1  0x000056390d5f39e7 in __gthread_cond_timedwait (__cond=0x56390f079bf0, __mutex=0x56390f0791d8, __abs_timeout=0x7f4a88ff8a50) at /usr/include/x86_64-linux-gnu/c++/9/bits/gthr-default.h:872
No locals.
#2  0x000056390d5f6615 in std::condition_variable::__wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x56390f079bf0, __lock=@0x7f4a88ff8b70: {_M_device = 0x56390f0791d8, _M_owns = true}, __atime=@0x7f4a88ff8ab0: {__d = {__r = 1674515379971660979}}) at /usr/include/c++/9/condition_variable:188
        __s = {__d = {__r = 1674515379}}
        __ns = {__r = 971660979}
        __ts = {tv_sec = 1674515379, tv_nsec = 971660979}
#3  0x000056390d5f5821 in std::condition_variable::wait_until<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x56390f079bf0, __lock=@0x7f4a88ff8b70: {_M_device = 0x56390f0791d8, _M_owns = true}, __atime=@0x7f4a88ff8b08: {__d = {__r = 43489316347939508}}) at /usr/include/c++/9/condition_variable:121
        __c_entry = {__d = {__r = 43489256347940149}}
        __s_entry = {__d = {__r = 1674515319971661620}}
        __delta = {__r = 59999999359}
        __s_atime = {__d = {__r = 1674515379971660979}}
#4  0x000056390d5f4aaa in std::condition_variable::wait_for<long, std::ratio<1l, 1000l> > (this=0x56390f079bf0, __lock=@0x7f4a88ff8b70: {_M_device = 0x56390f0791d8, _M_owns = true}, __rtime=@0x56390f079200: {__r = 60000}) at /usr/include/c++/9/condition_variable:152
        __reltime = {__r = 60000000000}
#5  0x000056390d5f2179 in tpool::thread_pool_generic::wait_for_tasks (this=0x56390f0790c0, lk=@0x7f4a88ff8b70: {_M_device = 0x56390f0791d8, _M_owns = true}, thread_data=0x56390f079bf0) at /home/buildbot/amd64-ubuntu-2004-debug/build/tpool/tpool_generic.cc:480
        __PRETTY_FUNCTION__ = "bool tpool::thread_pool_generic::wait_for_tasks(std::unique_lock<std::mutex>&, tpool::worker_data*)"
#6  0x000056390d5f2389 in tpool::thread_pool_generic::get_task (this=0x56390f0790c0, thread_var=0x56390f079bf0, t=0x7f4a88ff8bb8) at /home/buildbot/amd64-ubuntu-2004-debug/build/tpool/tpool_generic.cc:533
        lk = {_M_device = 0x56390f0791d8, _M_owns = true}
        __PRETTY_FUNCTION__ = "bool tpool::thread_pool_generic::get_task(tpool::worker_data*, tpool::task**)"
#7  0x000056390d5f2623 in tpool::thread_pool_generic::worker_main (this=0x56390f0790c0, thread_var=0x56390f079bf0) at /home/buildbot/amd64-ubuntu-2004-debug/build/tpool/tpool_generic.cc:578
        task = 0x56390f1db9e0
#8  0x000056390d5f8a36 in std::__invoke_impl<void, void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> (__f=@0x7f4ae8001348: (void (tpool::thread_pool_generic::*)(tpool::thread_pool_generic * const, tpool::worker_data *)) 0x56390d5f25be <tpool::thread_pool_generic::worker_main(tpool::worker_data*)>, __t=@0x7f4ae8001340: 0x56390f0790c0) at /usr/include/c++/9/bits/invoke.h:73
No locals.
#9  0x000056390d5f8918 in std::__invoke<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> (__fn=@0x7f4ae8001348: (void (tpool::thread_pool_generic::*)(tpool::thread_pool_generic * const, tpool::worker_data *)) 0x56390d5f25be <tpool::thread_pool_generic::worker_main(tpool::worker_data*)>) at /usr/include/c++/9/bits/invoke.h:95
No locals.
#10 0x000056390d5f8837 in std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> >::_M_invoke<0ul, 1ul, 2ul> (this=0x7f4ae8001338) at /usr/include/c++/9/thread:244
No locals.
#11 0x000056390d5f87d2 in std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> >::operator() (this=0x7f4ae8001338) at /usr/include/c++/9/thread:251
No locals.
#12 0x000056390d5f87b2 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> > >::_M_run (this=0x7f4ae8001330) at /usr/include/c++/9/thread:195
No locals.
#13 0x00007f4b0c82fde4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#14 0x00007f4b0c949609 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#15 0x00007f4b0c51a133 in clone () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
Thread 7 (Thread 0x7f4aeeffd700 (LWP 338968)):
#0  0x00007f4b0c9507d1 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#1  0x000056390d5f39e7 in __gthread_cond_timedwait (__cond=0x56390f0796f0, __mutex=0x56390f0791d8, __abs_timeout=0x7f4aeeffca50) at /usr/include/x86_64-linux-gnu/c++/9/bits/gthr-default.h:872
No locals.
#2  0x000056390d5f6615 in std::condition_variable::__wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x56390f0796f0, __lock=@0x7f4aeeffcb70: {_M_device = 0x56390f0791d8, _M_owns = true}, __atime=@0x7f4aeeffcab0: {__d = {__r = 1674515387999766749}}) at /usr/include/c++/9/condition_variable:188
        __s = {__d = {__r = 1674515387}}
        __ns = {__r = 999766749}
        __ts = {tv_sec = 1674515387, tv_nsec = 999766749}
#3  0x000056390d5f5821 in std::condition_variable::wait_until<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x56390f0796f0, __lock=@0x7f4aeeffcb70: {_M_device = 0x56390f0791d8, _M_owns = true}, __atime=@0x7f4aeeffcb08: {__d = {__r = 43489324376045278}}) at /usr/include/c++/9/condition_variable:121
        __c_entry = {__d = {__r = 43489264376045548}}
        __s_entry = {__d = {__r = 1674515327999767019}}
        __delta = {__r = 59999999730}
        __s_atime = {__d = {__r = 1674515387999766749}}
#4  0x000056390d5f4aaa in std::condition_variable::wait_for<long, std::ratio<1l, 1000l> > (this=0x56390f0796f0, __lock=@0x7f4aeeffcb70: {_M_device = 0x56390f0791d8, _M_owns = true}, __rtime=@0x56390f079200: {__r = 60000}) at /usr/include/c++/9/condition_variable:152
        __reltime = {__r = 60000000000}
#5  0x000056390d5f2179 in tpool::thread_pool_generic::wait_for_tasks (this=0x56390f0790c0, lk=@0x7f4aeeffcb70: {_M_device = 0x56390f0791d8, _M_owns = true}, thread_data=0x56390f0796f0) at /home/buildbot/amd64-ubuntu-2004-debug/build/tpool/tpool_generic.cc:480
        __PRETTY_FUNCTION__ = "bool tpool::thread_pool_generic::wait_for_tasks(std::unique_lock<std::mutex>&, tpool::worker_data*)"
#6  0x000056390d5f2389 in tpool::thread_pool_generic::get_task (this=0x56390f0790c0, thread_var=0x56390f0796f0, t=0x7f4aeeffcbb8) at /home/buildbot/amd64-ubuntu-2004-debug/build/tpool/tpool_generic.cc:533
        lk = {_M_device = 0x56390f0791d8, _M_owns = true}
        __PRETTY_FUNCTION__ = "bool tpool::thread_pool_generic::get_task(tpool::worker_data*, tpool::task**)"
#7  0x000056390d5f2623 in tpool::thread_pool_generic::worker_main (this=0x56390f0790c0, thread_var=0x56390f0796f0) at /home/buildbot/amd64-ubuntu-2004-debug/build/tpool/tpool_generic.cc:578
        task = 0x56390ed27460 <purge_coordinator_task>
#8  0x000056390d5f8a36 in std::__invoke_impl<void, void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> (__f=@0x7f4ae8001b18: (void (tpool::thread_pool_generic::*)(tpool::thread_pool_generic * const, tpool::worker_data *)) 0x56390d5f25be <tpool::thread_pool_generic::worker_main(tpool::worker_data*)>, __t=@0x7f4ae8001b10: 0x56390f0790c0) at /usr/include/c++/9/bits/invoke.h:73
No locals.
#9  0x000056390d5f8918 in std::__invoke<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> (__fn=@0x7f4ae8001b18: (void (tpool::thread_pool_generic::*)(tpool::thread_pool_generic * const, tpool::worker_data *)) 0x56390d5f25be <tpool::thread_pool_generic::worker_main(tpool::worker_data*)>) at /usr/include/c++/9/bits/invoke.h:95
No locals.
#10 0x000056390d5f8837 in std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> >::_M_invoke<0ul, 1ul, 2ul> (this=0x7f4ae8001b08) at /usr/include/c++/9/thread:244
No locals.
#11 0x000056390d5f87d2 in std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> >::operator() (this=0x7f4ae8001b08) at /usr/include/c++/9/thread:251
No locals.
#12 0x000056390d5f87b2 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> > >::_M_run (this=0x7f4ae8001b00) at /usr/include/c++/9/thread:195
No locals.
#13 0x00007f4b0c82fde4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#14 0x00007f4b0c949609 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#15 0x00007f4b0c51a133 in clone () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
Thread 6 (Thread 0x7f4b04131700 (LWP 314484)):
#0  0x00007f4b0c9507d1 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#1  0x000056390d69073b in safe_cond_timedwait (cond=0x56390e478bc0 <debug_sync_global+32>, mp=0x56390e478bf8 <debug_sync_global+88>, abstime=0x7f4b04130b10, file=0x56390dbfe040 "/home/buildbot/amd64-ubuntu-2004-debug/build/mysys/my_thr_init.c", line=609) at /home/buildbot/amd64-ubuntu-2004-debug/build/mysys/thr_mutex.c:548
        error = 32587
        save_state = {global = {__data = {__lock = 1, __count = 0, __owner = 314484, __nusers = 1, __kind = 2, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = "\001\000\000\000\000\000\000\000t\314\004\000\001\000\000\000\002", '\000' <repeats 22 times>, __align = 1}, mutex = {__data = {__lock = 2, __count = 0, __owner = 314484, __nusers = 1, __kind = 3, __spins = 25, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = "\002\000\000\000\000\000\000\000t\314\004\000\001\000\000\000\003\000\000\000\031", '\000' <repeats 18 times>, __align = 2}, file = 0x56390dbfe040 "/home/buildbot/amd64-ubuntu-2004-debug/build/mysys/my_thr_init.c", name = 0x56390d7d6ec0 "debug_sync_global.ds_mutex", line = 609, count = 0, create_flags = 0, active_flags = 0, id = 47, thread = 139960167634688, locked_mutex = 0x56390f1ff558, used_mutex = 0x56390f1ff5d8, prev = 0x0, next = 0x0}
#2  0x000056390d689ac5 in psi_cond_timedwait (that=0x56390e478bc0 <debug_sync_global+32>, mutex=0x56390e478bf8 <debug_sync_global+88>, abstime=0x7f4b04130b10, file=0x56390d7d6e80 "/home/buildbot/amd64-ubuntu-2004-debug/build/sql/debug_sync.cc", line=1480) at /home/buildbot/amd64-ubuntu-2004-debug/build/mysys/my_thr_init.c:609
        state = {m_flags = 7, m_operation = (PSI_COND_TIMEDWAIT | unknown: 32586), m_cond = 0x56390eef7280, m_mutex = 0x7f4b084209c0, m_thread = 0x7f4b0bc3ba40, m_timer_start = 108505078395908375, m_timer = 0x56390d143b26 <my_timer_cycles()>, m_wait = 0x7f4b0bc3c370}
        locker = 0x7f4b04130a40
        result = 0
#3  0x000056390cacc0fe in inline_mysql_cond_timedwait (that=0x56390e478bc0 <debug_sync_global+32>, mutex=0x56390e478bf8 <debug_sync_global+88>, abstime=0x7f4b04130b10, src_file=0x56390d7d6e80 "/home/buildbot/amd64-ubuntu-2004-debug/build/sql/debug_sync.cc", src_line=1480) at /home/buildbot/amd64-ubuntu-2004-debug/build/include/mysql/psi/mysql_thread.h:1086
No locals.
#4  0x000056390cacec97 in debug_sync_execute (thd=0x7f49a8000db8, action=0x7f49a8024b08) at /home/buildbot/amd64-ubuntu-2004-debug/build/sql/debug_sync.cc:1480
        old_mutex = 0x0
        old_cond = 0x0
        abstime = {tv_sec = 1674515490, tv_nsec = 443849000}
        restore_current_mutex = true
        error = 0
        old_proc_info = 0x56390d78f187 "Reset for next command"
        __PRETTY_FUNCTION__ = "void debug_sync_execute(THD*, st_debug_sync_action*)"
#5  0x000056390cacf121 in debug_sync (thd=0x7f49a8000db8, sync_point_name=0x56390dab7a91 "innobase_connection_closed", name_len=26) at /home/buildbot/amd64-ubuntu-2004-debug/build/sql/debug_sync.cc:1588
        ds_control = 0x7f49a800aeb8
        action = 0x7f49a8024b08
        __PRETTY_FUNCTION__ = "void debug_sync(THD*, const char*, size_t)"
#6  0x000056390d1e9693 in innobase_close_connection (hton=0x56390f078ab8, thd=0x7f49a8000db8) at /home/buildbot/amd64-ubuntu-2004-debug/build/storage/innobase/handler/ha_innodb.cc:4934
        trx = 0x7f4b068c6b80
        __PRETTY_FUNCTION__ = "int innobase_close_connection(handlerton*, THD*)"
#7  0x000056390ce2c528 in ha_close_connection (thd=0x7f49a8000db8) at /home/buildbot/amd64-ubuntu-2004-debug/build/sql/handler.cc:907
        hton = 0x56390f078ab8
        plugin = 0x7f49a80086b8
        i = 6
        __PRETTY_FUNCTION__ = "void ha_close_connection(THD*)"
#8  0x000056390ca00b9e in THD::free_connection (this=0x7f49a8000db8) at /home/buildbot/amd64-ubuntu-2004-debug/build/sql/sql_class.cc:1602
        __PRETTY_FUNCTION__ = "void THD::free_connection()"
#9  0x000056390c906fbf in unlink_thd (thd=0x7f49a8000db8) at /home/buildbot/amd64-ubuntu-2004-debug/build/sql/mysqld.cc:2780
No locals.
#10 0x000056390cc3f7c4 in do_handle_one_connection (connect=0x56390f71da98, put_in_cache=true) at /home/buildbot/amd64-ubuntu-2004-debug/build/sql/sql_connect.cc:1427
        create_user = true
        thr_create_utime = 43489126810783
        thd = 0x7f49a8000db8
#11 0x000056390cc3f4e8 in handle_one_connection (arg=0x56390f732b18) at /home/buildbot/amd64-ubuntu-2004-debug/build/sql/sql_connect.cc:1318
        connect = 0x56390f732b18
#12 0x000056390d1112bb in pfs_spawn_thread (arg=0x56390f4d9488) at /home/buildbot/amd64-ubuntu-2004-debug/build/storage/perfschema/pfs.cc:2201
        typed_arg = 0x56390f4d9488
        user_arg = 0x56390f732b18
        user_start_routine = 0x56390cc3f489 <handle_one_connection(void*)>
        pfs = 0x7f4b0bc3a3c0
        klass = 0x56390ee32c80
#13 0x00007f4b0c949609 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#14 0x00007f4b0c51a133 in clone () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
Thread 5 (Thread 0x7f4b068b4700 (LWP 307826)):
#0  0x00007f4b0c51373d in syscall () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
#1  0x000056390d5f94bc in my_getevents (ctx=0x7f4b068b5000, min_nr=1, nr=256, ev=0x7f4b068b1be0) at /home/buildbot/amd64-ubuntu-2004-debug/build/tpool/aio_linux.cc:62
        saved_errno = 0
        ret = 224368213
#2  0x000056390d5f96b1 in tpool::aio_linux::getevent_thread_routine (aio=0x56390ee50810) at /home/buildbot/amd64-ubuntu-2004-debug/build/tpool/aio_linux.cc:104
        ret = 1
        MAX_EVENTS = 256
        events = {{data = 0x0, obj = 0x56390f0be2a8, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0d96c8, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0d88b8, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0d8fc0, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0d8ef8, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0d9088, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0d79e0, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0dc288, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0d7c38, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0e0bf8, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0e0b30, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0d8980, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0e0cc0, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0dd2f0, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0da730, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0d9600, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0d9790, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0d8660, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0e09a0, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0e0810, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0e08d8, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0e0748, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0e0680, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0e05b8, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0e04f0, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0e0428, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0e0360, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0e0298, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0e01d0, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0e0108, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0e0040, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0dff78, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0dfeb0, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0dfde8, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0dfd20, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0dfc58, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0dfb90, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0dfac8, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0dfa00, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0df938, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0df870, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0df7a8, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0df6e0, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0df618, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0df550, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0df488, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0df3c0, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0df2f8, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0df230, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0df168, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0defd8, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0df0a0, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0def10, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0dee48, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0ded80, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0decb8, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0debf0, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0deb28, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0dea60, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0de998, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0de8d0, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0de808, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0de740, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0de678, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0de5b0, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0de4e8, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0de420, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0de358, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0de290, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0de1c8, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0de100, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0db6d0, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0d9218, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0d9920, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0d9858, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0d9f60, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0d9e98, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0ddea8, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0d9538, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0da280, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0d9dd0, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0ddde0, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0d8d68, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0de038, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0d9150, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0d9d08, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0d9b78, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0d99e8, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0d9c40, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0d7dc8, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0d8e30, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0d9ab0, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0d8bd8, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0dce40, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0dc990, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0dcf08, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0dd098, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0dca58, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0dd160, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0dd480, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0dcd78, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0dd868, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0d9470, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0dcbe8, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0db3b0, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0dcb20, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0dc8c8, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0dc738, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0dc800, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0dd228, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0dccb0, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0d8278, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0d8408, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0d8b10, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0dc5a8, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0da5a0, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0dc670, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0d7b70, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0d7e90, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0d7c38, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0dc288, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0d79e0, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0d9088, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0d8ef8, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0d8fc0, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0d88b8, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0d96c8, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0dcfd0, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0d8980, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0dd2f0, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0da730, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0d8660, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0d9600, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0d9790, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0d81b0, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0ddd18, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0d9538, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0dc1c0, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0ddea8, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0d87f0, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0d7f58, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0da028, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0da0f0, res = 16384, res2 = 0}, {data = 0x0, obj = 0x56390f0dc0f8, res = 16384, res2 = 0}, {data = 0x0, obj = 0x0, res = 0, res2 = 0} <repeats 111 times>, {data = 0x0, obj = 0x56390ef99ef0, res = 139960209062896, res2 = 94803064364776}}
#3  0x000056390d5fa1fe in std::__invoke_impl<void, void (*)(tpool::aio_linux*), tpool::aio_linux*> (__f=@0x56390ef99ef0: 0x56390d5f9655 <tpool::aio_linux::getevent_thread_routine(tpool::aio_linux*)>) at /usr/include/c++/9/bits/invoke.h:60
No locals.
#4  0x000056390d5fa15e in std::__invoke<void (*)(tpool::aio_linux*), tpool::aio_linux*> (__fn=@0x56390ef99ef0: 0x56390d5f9655 <tpool::aio_linux::getevent_thread_routine(tpool::aio_linux*)>) at /usr/include/c++/9/bits/invoke.h:95
No locals.
#5  0x000056390d5fa0bd in std::thread::_Invoker<std::tuple<void (*)(tpool::aio_linux*), tpool::aio_linux*> >::_M_invoke<0ul, 1ul> (this=0x56390ef99ee8) at /usr/include/c++/9/thread:244
No locals.
#6  0x000056390d5fa074 in std::thread::_Invoker<std::tuple<void (*)(tpool::aio_linux*), tpool::aio_linux*> >::operator() (this=0x56390ef99ee8) at /usr/include/c++/9/thread:251
No locals.
#7  0x000056390d5fa054 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (*)(tpool::aio_linux*), tpool::aio_linux*> > >::_M_run (this=0x56390ef99ee0) at /usr/include/c++/9/thread:195
No locals.
#8  0x00007f4b0c82fde4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#9  0x00007f4b0c949609 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#10 0x00007f4b0c51a133 in clone () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
Thread 4 (Thread 0x7f4b042a8700 (LWP 307859)):
#0  0x00007f4b0c950376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#1  0x000056390d690379 in safe_cond_wait (cond=0x56390e477620 <COND_manager>, mp=0x56390e477560 <LOCK_manager>, file=0x56390dbfe040 "/home/buildbot/amd64-ubuntu-2004-debug/build/mysys/my_thr_init.c", line=596) at /home/buildbot/amd64-ubuntu-2004-debug/build/mysys/thr_mutex.c:494
        error = 32587
        save_state = {global = {__data = {__lock = 1, __count = 0, __owner = 307859, __nusers = 1, __kind = 2, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = "\001\000\000\000\000\000\000\000\223\262\004\000\001\000\000\000\002", '\000' <repeats 22 times>, __align = 1}, mutex = {__data = {__lock = 1, __count = 0, __owner = 307859, __nusers = 1, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = "\001\000\000\000\000\000\000\000\223\262\004\000\001", '\000' <repeats 26 times>, __align = 1}, file = 0x56390d7c8b00 "/home/buildbot/amd64-ubuntu-2004-debug/build/sql/sql_manager.cc", name = 0x56390d7c8b8e "LOCK_manager", line = 117, count = 0, create_flags = 0, active_flags = 0, id = 83, thread = 139960169170688, locked_mutex = 0x7f4aa4000ff8, used_mutex = 0x7f4aa4001078, prev = 0x0, next = 0x0}
#2  0x000056390d689a12 in psi_cond_wait (that=0x56390e477620 <COND_manager>, mutex=0x56390e477560 <LOCK_manager>, file=0x56390d7c8b00 "/home/buildbot/amd64-ubuntu-2004-debug/build/sql/sql_manager.cc", line=103) at /home/buildbot/amd64-ubuntu-2004-debug/build/mysys/my_thr_init.c:596
        state = {m_flags = 7, m_operation = PSI_COND_WAIT, m_cond = 0x56390eef8180, m_mutex = 0x7f4b084268c0, m_thread = 0x7f4b0bb9e240, m_timer_start = 108504957746214675, m_timer = 0x56390d143b26 <my_timer_cycles()>, m_wait = 0x7f4b0bb9eb70}
        locker = 0x7f4b042a7be0
        result = 22073
#3  0x000056390ca78074 in inline_mysql_cond_wait (that=0x56390e477620 <COND_manager>, mutex=0x56390e477560 <LOCK_manager>, src_file=0x56390d7c8b00 "/home/buildbot/amd64-ubuntu-2004-debug/build/sql/sql_manager.cc", src_line=103) at /home/buildbot/amd64-ubuntu-2004-debug/build/include/mysql/psi/mysql_thread.h:1070
No locals.
#4  0x000056390ca78496 in handle_manager (arg=0x0) at /home/buildbot/amd64-ubuntu-2004-debug/build/sql/sql_manager.cc:103
        cb = 0x0
        error = 0
        abstime = {tv_sec = 94803069882512, tv_nsec = 688}
        reset_flush_time = true
#5  0x000056390d1112bb in pfs_spawn_thread (arg=0x56390f4dd0a8) at /home/buildbot/amd64-ubuntu-2004-debug/build/storage/perfschema/pfs.cc:2201
        typed_arg = 0x56390f4dd0a8
        user_arg = 0x0
        user_start_routine = 0x56390ca782f1 <handle_manager(void*)>
        pfs = 0x7f4b0bb9e240
        klass = 0x56390ee32b00
#6  0x00007f4b0c949609 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#7  0x00007f4b0c51a133 in clone () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
Thread 3 (Thread 0x7f4b074c7700 (LWP 307790)):
#0  0x00007f4b0c9507d1 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#1  0x000056390d69073b in safe_cond_timedwait (cond=0x56390ecacc80 <COND_checkpoint>, mp=0x56390ecacbc0 <LOCK_checkpoint>, abstime=0x7f4b074c6c10, file=0x56390dbfe040 "/home/buildbot/amd64-ubuntu-2004-debug/build/mysys/my_thr_init.c", line=609) at /home/buildbot/amd64-ubuntu-2004-debug/build/mysys/thr_mutex.c:548
        error = 32587
        save_state = {global = {__data = {__lock = 1, __count = 0, __owner = 307790, __nusers = 1, __kind = 2, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = "\001\000\000\000\000\000\000\000N\262\004\000\001\000\000\000\002", '\000' <repeats 22 times>, __align = 1}, mutex = {__data = {__lock = 1, __count = 0, __owner = 307790, __nusers = 1, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = "\001\000\000\000\000\000\000\000N\262\004\000\001", '\000' <repeats 26 times>, __align = 1}, file = 0x56390da15220 "/home/buildbot/amd64-ubuntu-2004-debug/build/storage/maria/ma_servicethread.c", name = 0x56390da1526e "control->LOCK_control", line = 99, count = 0, create_flags = 0, active_flags = 0, id = 18, thread = 139960221726464, locked_mutex = 0x7f4af8000ff8, used_mutex = 0x7f4af8001078, prev = 0x0, next = 0x0}
#2  0x000056390d689ac5 in psi_cond_timedwait (that=0x56390ecacc80 <COND_checkpoint>, mutex=0x56390ecacbc0 <LOCK_checkpoint>, abstime=0x7f4b074c6c10, file=0x56390da15220 "/home/buildbot/amd64-ubuntu-2004-debug/build/storage/maria/ma_servicethread.c", line=115) at /home/buildbot/amd64-ubuntu-2004-debug/build/mysys/my_thr_init.c:609
        state = {m_flags = 7, m_operation = (unknown: 32586), m_cond = 0x56390eef7a00, m_mutex = 0x7f4b084221c0, m_thread = 0x7f4b0bb863c0, m_timer_start = 108505403781717650, m_timer = 0x56390d143b26 <my_timer_cycles()>, m_wait = 0x7f4b0bb86cf0}
        locker = 0x7f4b074c6b60
        result = 32587
#3  0x000056390d0727ae in inline_mysql_cond_timedwait (that=0x56390ecacc80 <COND_checkpoint>, mutex=0x56390ecacbc0 <LOCK_checkpoint>, abstime=0x7f4b074c6c10, src_file=0x56390da15220 "/home/buildbot/amd64-ubuntu-2004-debug/build/storage/maria/ma_servicethread.c", src_line=115) at /home/buildbot/amd64-ubuntu-2004-debug/build/include/mysql/psi/mysql_thread.h:1086
No locals.
#4  0x000056390d072aed in my_service_thread_sleep (control=0x56390e237000 <checkpoint_control>, sleep_time=30000000000) at /home/buildbot/amd64-ubuntu-2004-debug/build/storage/maria/ma_servicethread.c:115
        abstime = {tv_sec = 1674515381, tv_nsec = 454148000}
        res = 0 '\000'
#5  0x000056390d067373 in ma_checkpoint_background (arg=0x1e) at /home/buildbot/amd64-ubuntu-2004-debug/build/storage/maria/ma_checkpoint.c:725
        interval = 30
        sleeps = 180
        sleep_time = 30
        log_horizon_at_last_checkpoint = 4299683382
        pagecache_flushes_at_last_checkpoint = 14441
        pages_bunch_size = 0
        filter_param = {up_to_lsn = 4299683192, max_pages = 688}
        dfile = 0x7f4af8072518
        kfile = 0x7f4af8071748
        __PRETTY_FUNCTION__ = "ma_checkpoint_background"
#6  0x000056390d1112bb in pfs_spawn_thread (arg=0x56390f061e68) at /home/buildbot/amd64-ubuntu-2004-debug/build/storage/perfschema/pfs.cc:2201
        typed_arg = 0x56390f061e68
        user_arg = 0x1e
        user_start_routine = 0x56390d06703a <ma_checkpoint_background>
        pfs = 0x7f4b0bb863c0
        klass = 0x56390ee337c0
#7  0x00007f4b0c949609 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#8  0x00007f4b0c51a133 in clone () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
Thread 2 (Thread 0x7f4b08040700 (LWP 307784)):
#0  0x00007f4b0c9507d1 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#1  0x000056390d69073b in safe_cond_timedwait (cond=0x56390ed3f6e0 <COND_timer>, mp=0x56390ed3f620 <LOCK_timer>, abstime=0x7f4b0803fc90, file=0x56390dbfe040 "/home/buildbot/amd64-ubuntu-2004-debug/build/mysys/my_thr_init.c", line=609) at /home/buildbot/amd64-ubuntu-2004-debug/build/mysys/thr_mutex.c:548
        error = 32587
        save_state = {global = {__data = {__lock = 1, __count = 0, __owner = 307784, __nusers = 1, __kind = 2, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = "\001\000\000\000\000\000\000\000H\262\004\000\001\000\000\000\002", '\000' <repeats 22 times>, __align = 1}, mutex = {__data = {__lock = 2, __count = 0, __owner = 307784, __nusers = 1, __kind = 3, __spins = 8, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = "\002\000\000\000\000\000\000\000H\262\004\000\001\000\000\000\003\000\000\000\b", '\000' <repeats 18 times>, __align = 2}, file = 0x56390dbfe040 "/home/buildbot/amd64-ubuntu-2004-debug/build/mysys/my_thr_init.c", name = 0x56390dbfeec0 "LOCK_timer", line = 609, count = 0, create_flags = 0, active_flags = 0, id = 12, thread = 139960233756416, locked_mutex = 0x7f4b000022d8, used_mutex = 0x7f4b00002358, prev = 0x0, next = 0x0}
#2  0x000056390d689ac5 in psi_cond_timedwait (that=0x56390ed3f6e0 <COND_timer>, mutex=0x56390ed3f620 <LOCK_timer>, abstime=0x7f4b0803fc90, file=0x56390dbfee80 "/home/buildbot/amd64-ubuntu-2004-debug/build/mysys/thr_timer.c", line=321) at /home/buildbot/amd64-ubuntu-2004-debug/build/mysys/my_thr_init.c:609
        state = {m_flags = 7, m_operation = PSI_COND_WAIT, m_cond = 0x56390eef7300, m_mutex = 0x7f4b08420cc0, m_thread = 0x7f4b0bb84d40, m_timer_start = 108505423460517100, m_timer = 0x56390d143b26 <my_timer_cycles()>, m_wait = 0x7f4b0bb85670}
        locker = 0x7f4b0803fbc0
        result = 22073
#3  0x000056390d6913bf in inline_mysql_cond_timedwait (that=0x56390ed3f6e0 <COND_timer>, mutex=0x56390ed3f620 <LOCK_timer>, abstime=0x7f4b0803fc90, src_file=0x56390dbfee80 "/home/buildbot/amd64-ubuntu-2004-debug/build/mysys/thr_timer.c", src_line=321) at /home/buildbot/amd64-ubuntu-2004-debug/build/include/mysql/psi/mysql_thread.h:1086
No locals.
#4  0x000056390d691e43 in timer_handler (arg=0x0) at /home/buildbot/amd64-ubuntu-2004-debug/build/mysys/thr_timer.c:321
        error = 110
        top_time = 0x56390f0792d8
        now = {tv_sec = 1674515359, tv_nsec = 341458000}
        abstime = {tv_sec = 1674515359, tv_nsec = 741479000}
#5  0x000056390d1112bb in pfs_spawn_thread (arg=0x56390ef9db78) at /home/buildbot/amd64-ubuntu-2004-debug/build/storage/perfschema/pfs.cc:2201
        typed_arg = 0x56390ef9db78
        user_arg = 0x0
        user_start_routine = 0x56390d691cfd <timer_handler>
        pfs = 0x7f4b0bb84d40
        klass = 0x56390ee33040
#6  0x00007f4b0c949609 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#7  0x00007f4b0c51a133 in clone () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
Thread 1 (Thread 0x7f4b0c25f940 (LWP 307712)):
#0  0x00007f4b0c951f68 in pthread_kill () from /lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#1  0x000056390d68b39e in my_write_core (sig=6) at /home/buildbot/amd64-ubuntu-2004-debug/build/mysys/stacktrace.c:424
No locals.
#2  0x000056390ce2871f in handle_fatal_signal (sig=6) at /home/buildbot/amd64-ubuntu-2004-debug/build/sql/signal_handler.cc:357
        curr_time = 1674515358
        tm = {tm_sec = 18, tm_min = 9, tm_hour = 23, tm_mday = 23, tm_mon = 0, tm_year = 123, tm_wday = 1, tm_yday = 22, tm_isdst = 0, tm_gmtoff = 0, tm_zone = 0x56390ee34f70 "UTC"}
        thd = 0x0
        print_invalid_query_pointer = false
#3  <signal handler called>
No symbol table info available.
#4  0x00007f4b0c50ffcb in select () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
#5  0x000056390d687816 in my_sleep (m_seconds=1000) at /home/buildbot/amd64-ubuntu-2004-debug/build/mysys/my_sleep.c:29
        t = {tv_sec = 0, tv_usec = 784}
#6  0x000056390c904e87 in close_connections () at /home/buildbot/amd64-ubuntu-2004-debug/build/sql/mysqld.cc:1837
        n_threads_awaiting_ack = 0
#7  0x000056390c90ce8d in mysqld_main (argc=156, argv=0x56390ee16dd0) at /home/buildbot/amd64-ubuntu-2004-debug/build/sql/mysqld.cc:5955
        please_close_stdin = true
        ho_error = 0
        new_thread_stack_size = 299008
        user = 0x7f4a402c8d68 '\217' <repeats 44 times>
#8  0x000056390c9019ed in main (argc=23, argv=0x7ffe54d0e2e8) at /home/buildbot/amd64-ubuntu-2004-debug/build/sql/main.cc:34
No locals.

Comment by Marko Mäkelä [ 2023-01-24 ]

Another failure in a 10.9 based branch:

main.kill_debug 'innodb'                 w17 [ pass ]    156
main.statistics_upgrade_not_done 'innodb' w17 [ fail ]  Found warnings/errors in server log file!
#5  0x00005556ecea9c81 in debug_sync (thd=0x7f9418000db8, sync_point_name=0x5556ede81771 "innobase_connection_closed", name_len=26) at /home/buildbot/amd64-ubuntu-2004-debug/build/sql/debug_sync.cc:1588
        ds_control = 0x7f941800aea8
        action = 0x7f941801a118
        __PRETTY_FUNCTION__ = "void debug_sync(THD*, const char*, size_t)"
#6  0x00005556ed5bf2bf in innobase_close_connection (hton=0x5556efd52298, thd=0x7f9418000db8) at /home/buildbot/amd64-ubuntu-2004-debug/build/storage/innobase/handler/ha_innodb.cc:4951

Comment by Marko Mäkelä [ 2023-01-25 ]

A failure in a 11.0 based branch:

main.kill_debug 'innodb'                 w6 [ pass ]    175
main.statistics_upgrade_not_done 'innodb' w6 [ fail ]  Found warnings/errors in server log file!
#5  0x0000564989c2c0c1 in debug_sync (thd=0x7f4278000db8, sync_point_name=0x56498ac04061 "innobase_connection_closed", name_len=26) at /home/buildbot/amd64-ubuntu-2004-debug/build/sql/debug_sync.cc:1588
        ds_control = 0x7f427800aec8
        action = 0x7f4278024c68
        __PRETTY_FUNCTION__ = "void debug_sync(THD*, const char*, size_t)"
#6  0x000056498a34701b in innobase_close_connection (hton=0x56498d3e4858, thd=0x7f4278000db8) at /home/buildbot/amd64-ubuntu-2004-debug/build/storage/innobase/handler/ha_innodb.cc:4881

Comment by Marko Mäkelä [ 2023-03-06 ]

This is still causing trouble. A failure in a 10.8 based branch:

main.kill_debug 'innodb'                 w17 [ pass ]    152
main.statistics_upgrade_not_done 'innodb' w17 [ fail ]  Found warnings/errors in server log file!
#5  0x00005634a4daee55 in debug_sync (thd=0x7f25f8000db8, sync_point_name=0x5634a5d73eb1 "innobase_connection_closed", name_len=26) at /home/buildbot/amd64-ubuntu-2004-debug/build/sql/debug_sync.cc:1760
        ds_control = 0x7f25f800ae88
        action = 0x7f25f801a108
        __PRETTY_FUNCTION__ = "void debug_sync(THD*, const char*, size_t)"
#6  0x00005634a54b880d in innobase_close_connection (hton=0x5634a7a7d148, thd=0x7f25f8000db8) at /home/buildbot/amd64-ubuntu-2004-debug/build/storage/innobase/handler/ha_innodb.cc:4949
        trx = 0x7f26aac60b80
        __PRETTY_FUNCTION__ = "int innobase_close_connection(handlerton*, THD*)"

Comment by Otto Kekäläinen [ 2023-03-24 ]

Potentially related, seen on Debian buildd for s390x:

main.statistics_upgrade_not_done 'innodb' w2 [ fail ]  Found warnings/errors in server log file!
        Test ended at 2023-03-18 21:22:57
line
Attempting backtrace. You can use the following information to find out
^ Found warnings in /<<PKGBUILDDIR>>/builddir/mysql-test/var/2/log/mysqld.1.err
ok
 
 - found 'core' (0/1)
Core generated by '/<<PKGBUILDDIR>>/builddir/sql/mariadbd'
Output from gdb follows. The first stack trace is from the failing thread.
The following stack traces are from all threads (so the failing one is
duplicated).
--------------------------
[New LWP 309050]
[New LWP 310004]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/s390x-linux-gnu/libthread_db.so.1".
Core was generated by `/<<PKGBUILDDIR>>/builddir/sql/mariadbd --defaults-group-su'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x000003ff8e4c8992 in kill () from /lib/s390x-linux-gnu/libc.so.6
[Current thread is 1 (Thread 0x3ff8f3ea820 (LWP 309050))]
#0  0x000003ff8e4c8992 in kill () from /lib/s390x-linux-gnu/libc.so.6
#1  0x000002aa0891734a in handle_fatal_signal (sig=<optimized out>) at ./sql/signal_handler.cc:369
#2  <signal handler called>
#3  0x000002aa08df9512 in my_read (Filedes=<optimized out>, Buffer=0x4 <error: Cannot access memory at address 0x4>, Count=4096, MyFlags=<optimized out>) at ./mysys/my_read.c:63
#4  0x000002aa08916de6 in output_core_info () at ./sql/signal_handler.cc:91
#5  0x000002aa08917318 in handle_fatal_signal (sig=<optimized out>) at ./sql/signal_handler.cc:353
#6  <signal handler called>
#7  0x000003ff8e58e632 in munmap () from /lib/s390x-linux-gnu/libc.so.6
#8  0x000003ff8e527790 in free () from /lib/s390x-linux-gnu/libc.so.6
#9  0x000002aa08b47ed2 in aligned_free (ptr=<optimized out>) at ./include/aligned.h:37
#10 pfs_free (ptr=<optimized out>, size=3145728, klass=0x2aa0a0c6200 <builtin_memory_file>) at ./storage/perfschema/pfs_global.cc:83
#11 pfs_free (ptr=<optimized out>, size=3145728, klass=0x2aa0a0c6200 <builtin_memory_file>) at ./storage/perfschema/pfs_global.cc:78
#12 pfs_free_array (klass=0x2aa0a0c6200 <builtin_memory_file>, n=<optimized out>, size=size@entry=768, ptr=<optimized out>) at ./storage/perfschema/pfs_global.cc:134
#13 0x000002aa08b4a300 in PFS_buffer_default_allocator<PFS_file>::free_array (array=0x2aa0c1bc660, this=<optimized out>) at ./storage/perfschema/pfs_buffer_container.h:164
#14 PFS_buffer_scalable_container<PFS_file, 4096, 4096, PFS_buffer_default_array<PFS_file>, PFS_buffer_default_allocator<PFS_file> >::cleanup (this=<optimized out>) at ./storage/perfschema/pfs_buffer_container.h:506
#15 PFS_buffer_scalable_container<PFS_file, 4096, 4096, PFS_buffer_default_array<PFS_file>, PFS_buffer_default_allocator<PFS_file> >::cleanup (this=<optimized out>) at ./storage/perfschema/pfs_buffer_container.h:491
#16 cleanup_instruments () at ./storage/perfschema/pfs_instr.cc:222
#17 0x000002aa08b57ebc in cleanup_performance_schema () at ./storage/perfschema/pfs_server.cc:296
#18 0x000002aa08b583a0 in shutdown_performance_schema () at ./storage/perfschema/pfs_server.cc:326
#19 0x000002aa085c4c92 in mysqld_exit (exit_code=exit_code@entry=0) at ./sql/mysqld.cc:1945
#20 0x000002aa085cf93e in mysqld_main (argc=<optimized out>, argv=<optimized out>) at ./sql/mysqld.cc:6035
#21 0x000003ff8e4ab84a in ?? () from /lib/s390x-linux-gnu/libc.so.6
#22 0x000003ff8e4ab932 in __libc_start_main () from /lib/s390x-linux-gnu/libc.so.6
#23 0x000002aa085c27d0 in _start ()
 
Thread 2 (LWP 310004):
#0  0x000000009d4e96c8 in ?? ()
No symbol table info available.
#1  0x0000000000000000 in ?? ()
No symbol table info available.
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
 
Thread 1 (Thread 0x3ff8f3ea820 (LWP 309050)):
#0  0x000003ff8e4c8992 in kill () from /lib/s390x-linux-gnu/libc.so.6
No symbol table info available.
#1  0x000002aa0891734a in handle_fatal_signal (sig=<optimized out>) at ./sql/signal_handler.cc:369
        curr_time = 4294967296
        tm = {tm_sec = 1023, tm_min = -687894235, tm_hour = 0, tm_mday = 0, tm_mon = 1, tm_year = 18117, tm_wday = 682, tm_yday = 159019104, tm_isdst = 1023, tm_gmtoff = 4397358616184, tm_zone = 0xfb00000000 <error: Cannot access memory at address 0xfb00000000>}
        thd = <optimized out>
        print_invalid_query_pointer = <optimized out>
#2  <signal handler called>
No symbol table info available.
#3  0x000002aa08df9512 in my_read (Filedes=<optimized out>, Buffer=0x4 <error: Cannot access memory at address 0x4>, Count=4096, MyFlags=<optimized out>) at ./mysys/my_read.c:63
        got_errno = <optimized out>
        readbytes = 185
        save_count = <optimized out>
#4  0x000002aa08916de6 in output_core_info () at ./sql/signal_handler.cc:91
        buff = "Linux version 5.10.0-21-s390x (debian-kernel@lists.debian.org) (gcc-10 (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2) #1 SMP Debian 5.10.162-1 (2023-01-21)\nnlimited", ' ' <repeats 12 times>...
        len = <optimized out>
        fd = <optimized out>
#5  0x000002aa08917318 in handle_fatal_signal (sig=<optimized out>) at ./sql/signal_handler.cc:353
        curr_time = 1679174575
        tm = {tm_sec = 55, tm_min = 22, tm_hour = 21, tm_mday = 18, tm_mon = 2, tm_year = 123, tm_wday = 6, tm_yday = 76, tm_isdst = 0, tm_gmtoff = 0, tm_zone = 0x2aa0c0883f0 "UTC"}
        thd = 0x0
        print_invalid_query_pointer = 95
#6  <signal handler called>
No symbol table info available.
#7  0x000003ff8e58e632 in munmap () from /lib/s390x-linux-gnu/libc.so.6
No symbol table info available.
#8  0x000003ff8e527790 in free () from /lib/s390x-linux-gnu/libc.so.6
No symbol table info available.
#9  0x000002aa08b47ed2 in aligned_free (ptr=<optimized out>) at ./include/aligned.h:37
No locals.
#10 pfs_free (ptr=<optimized out>, size=3145728, klass=0x2aa0a0c6200 <builtin_memory_file>) at ./storage/perfschema/pfs_global.cc:83
No locals.
#11 pfs_free (ptr=<optimized out>, size=3145728, klass=0x2aa0a0c6200 <builtin_memory_file>) at ./storage/perfschema/pfs_global.cc:78
No locals.
#12 pfs_free_array (klass=0x2aa0a0c6200 <builtin_memory_file>, n=<optimized out>, size=size@entry=768, ptr=<optimized out>) at ./storage/perfschema/pfs_global.cc:134
        array_size = 3145728
        __PRETTY_FUNCTION__ = "void pfs_free_array(PFS_builtin_memory_class*, size_t, size_t, void*)"
#13 0x000002aa08b4a300 in PFS_buffer_default_allocator<PFS_file>::free_array (array=0x2aa0c1bc660, this=<optimized out>) at ./storage/perfschema/pfs_buffer_container.h:164
        __PRETTY_FUNCTION__ = <optimized out>
#14 PFS_buffer_scalable_container<PFS_file, 4096, 4096, PFS_buffer_default_array<PFS_file>, PFS_buffer_default_allocator<PFS_file> >::cleanup (this=<optimized out>) at ./storage/perfschema/pfs_buffer_container.h:506
        i = <optimized out>
        page = 0x2aa0c1bc660
        i = <optimized out>
        page = <optimized out>
#15 PFS_buffer_scalable_container<PFS_file, 4096, 4096, PFS_buffer_default_array<PFS_file>, PFS_buffer_default_allocator<PFS_file> >::cleanup (this=<optimized out>) at ./storage/perfschema/pfs_buffer_container.h:491
        i = <optimized out>
        page = <optimized out>
        i = <optimized out>
        page = <optimized out>
#16 cleanup_instruments () at ./storage/perfschema/pfs_instr.cc:222
No locals.
#17 0x000002aa08b57ebc in cleanup_performance_schema () at ./storage/perfschema/pfs_server.cc:296
No locals.
#18 0x000002aa08b583a0 in shutdown_performance_schema () at ./storage/perfschema/pfs_server.cc:326
No locals.
#19 0x000002aa085c4c92 in mysqld_exit (exit_code=exit_code@entry=0) at ./sql/mysqld.cc:1945
No locals.
#20 0x000002aa085cf93e in mysqld_main (argc=<optimized out>, argv=<optimized out>) at ./sql/mysqld.cc:6035
        please_close_stdin = true
        ho_error = <optimized out>
        new_thread_stack_size = <optimized out>
        user = <optimized out>
#21 0x000003ff8e4ab84a in ?? () from /lib/s390x-linux-gnu/libc.so.6
No symbol table info available.
#22 0x000003ff8e4ab932 in __libc_start_main () from /lib/s390x-linux-gnu/libc.so.6
No symbol table info available.
#23 0x000002aa085c27d0 in _start ()
No symbol table info available.

https://buildd.debian.org/status/fetch.php?pkg=mariadb&arch=s390x&ver=1%3A10.11.2-2%7Eexp1&stamp=1679174792&raw=0

Comment by Marko Mäkelä [ 2023-05-15 ]

otto, the failure that you posted should be unrelated and worth a separate bug ticket. The log that you linked to includes the following:

main.kill_debug 'innodb'                 w2 [ skipped ]  debug_sync is not available

This hang is due to misuse of DEBUG_SYNC in the test main.kill_debug, which will cause shutdown to hang in a further test that will be executed on the same worker. One more victim is the test sql_sequence.read_only, as noted in MDEV-12700.

Comment by Marko Mäkelä [ 2023-06-28 ]

I am restoring the priority of this to Critical, because I think that this fails on some mandatory staging builder at a probability of ⅓ or more.

Generated at Thu Feb 08 10:13:32 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.