Details
-
Bug
-
Status: Closed (View Workflow)
-
Blocker
-
Resolution: Fixed
-
10.6
Description
Notes: Set to a blocker, marked as regression and linked as caused by MDEV-26827 assuming it to be a sibling of MDEV-31084. Feel free to unset if it isn't.
A fix for MDEV-31084 is already in the branch.
|
10.6 51e62cb3b3b5f28a67c3c5862b3cb60ecf87180a |
mysqld: /data/MDEV-31084/10.6c/tpool/tpool_generic.cc:874: virtual void tpool::thread_pool_generic::wait_begin(): Assertion `!tls_worker_data->is_waiting()' failed.
|
230423 10:07:53 [ERROR] mysqld got signal 6 ;
|
#3 0x0000558275e8efd6 in __GI___assert_fail (assertion=0x55725d0ea148 "!tls_worker_data->is_waiting()",
|
file=0x55725d0e9e10 "/data/MDEV-31084/10.6c/tpool/tpool_generic.cc", line=874,
|
function=0x55725d0ea168 "virtual void tpool::thread_pool_generic::wait_begin()") at assert.c:101
|
#4 0x000055725caaeb65 in tpool::thread_pool_generic::wait_begin (this=0x55725f2f8500)
|
at /data/MDEV-31084/10.6c/tpool/tpool_generic.cc:874
|
#5 0x000055725cab4a6f in tpool::tpool_wait_begin () at /data/MDEV-31084/10.6c/tpool/wait_notification.cc:15
|
#6 0x000055725c7d3aac in os_aio_wait_until_no_pending_writes_low () at /data/MDEV-31084/10.6c/storage/innobase/os/os0file.cc:3652
|
#7 0x000055725c7d3ad6 in os_aio_wait_until_no_pending_writes () at /data/MDEV-31084/10.6c/storage/innobase/os/os0file.cc:3663
|
#8 0x000055725c9b843e in buf_flush_wait (lsn=65397942) at /data/MDEV-31084/10.6c/storage/innobase/buf/buf0flu.cc:1865
|
#9 0x000055725c9b8930 in buf_flush_wait_flushed (sync_lsn=65397942) at /data/MDEV-31084/10.6c/storage/innobase/buf/buf0flu.cc:1911
|
#10 0x000055725c78ef45 in log_checkpoint_margin () at /data/MDEV-31084/10.6c/storage/innobase/log/log0log.cc:1046
|
#11 0x000055725c78ef84 in log_check_margins () at /data/MDEV-31084/10.6c/storage/innobase/log/log0log.cc:1062
|
#12 0x000055725c81479b in log_free_check () at /data/MDEV-31084/10.6c/storage/innobase/include/log0log.inl:309
|
#13 0x000055725c815f71 in que_run_threads_low (thr=0x55725f3c5370) at /data/MDEV-31084/10.6c/storage/innobase/que/que0que.cc:639
|
#14 0x000055725c816095 in que_run_threads (thr=0x55725f3c5370) at /data/MDEV-31084/10.6c/storage/innobase/que/que0que.cc:664
|
#15 0x000055725c8f5e7b in trx_purge (n_tasks=1, truncate=false) at /data/MDEV-31084/10.6c/storage/innobase/trx/trx0purge.cc:1359
|
#16 0x000055725c8e0630 in purge_coordinator_state::do_purge (this=0x55725e0c1b00 <purge_state>)
|
at /data/MDEV-31084/10.6c/storage/innobase/srv/srv0srv.cc:1692
|
#17 0x000055725c8df634 in purge_coordinator_callback () at /data/MDEV-31084/10.6c/storage/innobase/srv/srv0srv.cc:1846
|
#18 0x000055725cab42ac in tpool::task_group::execute (this=0x55725e0c1d40 <purge_coordinator_task_group>,
|
t=0x55725e0c1de0 <purge_coordinator_task>) at /data/MDEV-31084/10.6c/tpool/task_group.cc:55
|
#19 0x000055725cab45e6 in tpool::task::execute (this=0x55725e0c1de0 <purge_coordinator_task>)
|
at /data/MDEV-31084/10.6c/tpool/task.cc:32
|
#20 0x000055725caadc11 in tpool::thread_pool_generic::worker_main (this=0x55725f2f8500, thread_var=0x55725f2f8a30)
|
at /data/MDEV-31084/10.6c/tpool/tpool_generic.cc:580
|
#21 0x000055725cab40b4 in std::__invoke_impl<void, void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> (__f=
|
@0x79c48001a9a8: (void (tpool::thread_pool_generic::*)(class tpool::thread_pool_generic * const, struct tpool::worker_data *)) 0x55725caadb78 <tpool::thread_pool_generic::worker_main(tpool::worker_data*)>, __t=@0x79c48001a9a0: 0x55725f2f8500)
|
at /usr/include/c++/9/bits/invoke.h:73
|
#22 0x000055725cab3f96 in std::__invoke<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> (__fn=
|
@0x79c48001a9a8: (void (tpool::thread_pool_generic::*)(class tpool::thread_pool_generic * const, struct tpool::worker_data *)) 0x55725caadb78 <tpool::thread_pool_generic::worker_main(tpool::worker_data*)>) at /usr/include/c++/9/bits/invoke.h:95
|
#23 0x000055725cab3eb5 in std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> >::_M_invoke<0ul, 1ul, 2ul> (this=0x79c48001a998) at /usr/include/c++/9/thread:244
|
#24 0x000055725cab3e50 in std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> >::operator() (this=0x79c48001a998) at /usr/include/c++/9/thread:251
|
#25 0x000055725cab3e30 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> > >::_M_run (this=0x79c48001a990) at /usr/include/c++/9/thread:195
|
#26 0x0000000067ef4de4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
|
#27 0x0000558276055609 in start_thread (arg=<optimized out>) at pthread_create.c:477
|
#28 0x0000558275f7a133 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
|
Attachments
Issue Links
- is caused by
-
-
- Closed
-
- relates to
-
-
- Closed
-
Activity
| Field | Original Value | New Value |
|---|---|---|
| Link |
This issue is caused by |
| Description |
_Notes: Set to a blocker, marked as regression and linked as caused by A fix for {noformat:title=10.6 51e62cb3b3b5f28a67c3c5862b3cb60ecf87180a} mysqld: /data/ 230423 10:07:53 [ERROR] mysqld got signal 6 ; {noformat} {noformat} #3 0x0000558275e8efd6 in __GI___assert_fail (assertion=0x55725d0ea148 "!tls_worker_data->is_waiting()", file=0x55725d0e9e10 "/data/ function=0x55725d0ea168 "virtual void tpool::thread_pool_generic::wait_begin()") at assert.c:101 #4 0x000055725caaeb65 in tpool::thread_pool_generic::wait_begin (this=0x55725f2f8500) at /data/ #5 0x000055725cab4a6f in tpool::tpool_wait_begin () at /data/ #6 0x000055725c7d3aac in os_aio_wait_until_no_pending_writes_low () at /data/ #7 0x000055725c7d3ad6 in os_aio_wait_until_no_pending_writes () at /data/ #8 0x000055725c9b843e in buf_flush_wait (lsn=65397942) at /data/ #9 0x000055725c9b8930 in buf_flush_wait_flushed (sync_lsn=65397942) at /data/ #10 0x000055725c78ef45 in log_checkpoint_margin () at /data/ #11 0x000055725c78ef84 in log_check_margins () at /data/ #12 0x000055725c81479b in log_free_check () at /data/ #13 0x000055725c815f71 in que_run_threads_low (thr=0x55725f3c5370) at /data/ #14 0x000055725c816095 in que_run_threads (thr=0x55725f3c5370) at /data/ #15 0x000055725c8f5e7b in trx_purge (n_tasks=1, truncate=false) at /data/ #16 0x000055725c8e0630 in purge_coordinator_state::do_purge (this=0x55725e0c1b00 <purge_state>) at /data/ #17 0x000055725c8df634 in purge_coordinator_callback () at /data/ #18 0x000055725cab42ac in tpool::task_group::execute (this=0x55725e0c1d40 <purge_coordinator_task_group>, t=0x55725e0c1de0 <purge_coordinator_task>) at /data/ #19 0x000055725cab45e6 in tpool::task::execute (this=0x55725e0c1de0 <purge_coordinator_task>) at /data/ #20 0x000055725caadc11 in tpool::thread_pool_generic::worker_main (this=0x55725f2f8500, thread_var=0x55725f2f8a30) at /data/ #21 0x000055725cab40b4 in std::__invoke_impl<void, void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> (__f= @0x79c48001a9a8: (void (tpool::thread_pool_generic::*)(class tpool::thread_pool_generic * const, struct tpool::worker_data *)) 0x55725caadb78 <tpool::thread_pool_generic::worker_main(tpool::worker_data*)>, __t=@0x79c48001a9a0: 0x55725f2f8500) at /usr/include/c++/9/bits/invoke.h:73 #22 0x000055725cab3f96 in std::__invoke<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> (__fn= @0x79c48001a9a8: (void (tpool::thread_pool_generic::*)(class tpool::thread_pool_generic * const, struct tpool::worker_data *)) 0x55725caadb78 <tpool::thread_pool_generic::worker_main(tpool::worker_data*)>) at /usr/include/c++/9/bits/invoke.h:95 #23 0x000055725cab3eb5 in std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> >::_M_invoke<0ul, 1ul, 2ul> (this=0x79c48001a998) at /usr/include/c++/9/thread:244 #24 0x000055725cab3e50 in std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> >::operator() (this=0x79c48001a998) at /usr/include/c++/9/thread:251 #25 0x000055725cab3e30 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> > >::_M_run (this=0x79c48001a990) at /usr/include/c++/9/thread:195 #26 0x0000000067ef4de4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6 #27 0x0000558276055609 in start_thread (arg=<optimized out>) at pthread_create.c:477 #28 0x0000558275f7a133 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 {noformat} |
_Notes: Set to a blocker, marked as regression and linked as caused by _A fix for {noformat:title=10.6 51e62cb3b3b5f28a67c3c5862b3cb60ecf87180a} mysqld: /data/ 230423 10:07:53 [ERROR] mysqld got signal 6 ; {noformat} {noformat} #3 0x0000558275e8efd6 in __GI___assert_fail (assertion=0x55725d0ea148 "!tls_worker_data->is_waiting()", file=0x55725d0e9e10 "/data/ function=0x55725d0ea168 "virtual void tpool::thread_pool_generic::wait_begin()") at assert.c:101 #4 0x000055725caaeb65 in tpool::thread_pool_generic::wait_begin (this=0x55725f2f8500) at /data/ #5 0x000055725cab4a6f in tpool::tpool_wait_begin () at /data/ #6 0x000055725c7d3aac in os_aio_wait_until_no_pending_writes_low () at /data/ #7 0x000055725c7d3ad6 in os_aio_wait_until_no_pending_writes () at /data/ #8 0x000055725c9b843e in buf_flush_wait (lsn=65397942) at /data/ #9 0x000055725c9b8930 in buf_flush_wait_flushed (sync_lsn=65397942) at /data/ #10 0x000055725c78ef45 in log_checkpoint_margin () at /data/ #11 0x000055725c78ef84 in log_check_margins () at /data/ #12 0x000055725c81479b in log_free_check () at /data/ #13 0x000055725c815f71 in que_run_threads_low (thr=0x55725f3c5370) at /data/ #14 0x000055725c816095 in que_run_threads (thr=0x55725f3c5370) at /data/ #15 0x000055725c8f5e7b in trx_purge (n_tasks=1, truncate=false) at /data/ #16 0x000055725c8e0630 in purge_coordinator_state::do_purge (this=0x55725e0c1b00 <purge_state>) at /data/ #17 0x000055725c8df634 in purge_coordinator_callback () at /data/ #18 0x000055725cab42ac in tpool::task_group::execute (this=0x55725e0c1d40 <purge_coordinator_task_group>, t=0x55725e0c1de0 <purge_coordinator_task>) at /data/ #19 0x000055725cab45e6 in tpool::task::execute (this=0x55725e0c1de0 <purge_coordinator_task>) at /data/ #20 0x000055725caadc11 in tpool::thread_pool_generic::worker_main (this=0x55725f2f8500, thread_var=0x55725f2f8a30) at /data/ #21 0x000055725cab40b4 in std::__invoke_impl<void, void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> (__f= @0x79c48001a9a8: (void (tpool::thread_pool_generic::*)(class tpool::thread_pool_generic * const, struct tpool::worker_data *)) 0x55725caadb78 <tpool::thread_pool_generic::worker_main(tpool::worker_data*)>, __t=@0x79c48001a9a0: 0x55725f2f8500) at /usr/include/c++/9/bits/invoke.h:73 #22 0x000055725cab3f96 in std::__invoke<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> (__fn= @0x79c48001a9a8: (void (tpool::thread_pool_generic::*)(class tpool::thread_pool_generic * const, struct tpool::worker_data *)) 0x55725caadb78 <tpool::thread_pool_generic::worker_main(tpool::worker_data*)>) at /usr/include/c++/9/bits/invoke.h:95 #23 0x000055725cab3eb5 in std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> >::_M_invoke<0ul, 1ul, 2ul> (this=0x79c48001a998) at /usr/include/c++/9/thread:244 #24 0x000055725cab3e50 in std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> >::operator() (this=0x79c48001a998) at /usr/include/c++/9/thread:251 #25 0x000055725cab3e30 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> > >::_M_run (this=0x79c48001a990) at /usr/include/c++/9/thread:195 #26 0x0000000067ef4de4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6 #27 0x0000558276055609 in start_thread (arg=<optimized out>) at pthread_create.c:477 #28 0x0000558275f7a133 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 {noformat} |
| Link |
This issue relates to |
| Status | Open [ 1 ] | In Progress [ 3 ] |
| Labels | regression rr-profile | regression rr-profile-analyzed |
| Status | In Progress [ 3 ] | In Testing [ 10301 ] |
| Assignee | Marko Mäkelä [ marko ] | Elena Stepanova [ elenst ] |
| issue.field.resolutiondate | 2023-04-24 12:53:24.0 | 2023-04-24 12:53:24.862 |
| Fix Version/s | 10.6.13 [ 28514 ] | |
| Fix Version/s | 10.9.6 [ 28520 ] | |
| Fix Version/s | 10.10.4 [ 28522 ] | |
| Fix Version/s | 10.11.3 [ 28524 ] | |
| Fix Version/s | 11.1.1 [ 28704 ] | |
| Fix Version/s | 10.8.8 [ 28518 ] | |
| Fix Version/s | 11.0.2 [ 28706 ] | |
| Fix Version/s | 10.6 [ 24028 ] | |
| Assignee | Elena Stepanova [ elenst ] | Marko Mäkelä [ marko ] |
| Resolution | Fixed [ 1 ] | |
| Status | In Testing [ 10301 ] | Closed [ 6 ] |
The assertion fails, because a wait is being declared twice in the thread pool. The first time was around the buf_flush_wait() call:
thd_wait_begin(nullptr, THD_WAIT_DISKIO);tpool::tpool_wait_begin();buf_flush_wait(sync_lsn);tpool::tpool_wait_end();thd_wait_end(nullptr);The second time is inside os_aio_wait_until_no_pending_writes(), which is being called from buf_flush_wait().
This regression was caused by the removal of ‘double bookkeeping’ of pending writes and reads in InnoDB. We now rely solely on the read_slots and write_slots. I think that we must move the wait instrumentation from inside os_aio_wait_until_no_pending_writes() and os_aio_wait_until_no_pending_reads() to those callers that require it.