[MDEV-33278] Assertion failure in thd_get_thread_id at lock_wait_wsrep Created: 2024-01-19  Updated: 2024-01-30

Status: In Review
Project: MariaDB Server
Component/s: Galera, Storage Engine - InnoDB
Affects Version/s: 10.6
Fix Version/s: 10.6

Type: Bug Priority: Critical
Reporter: Jan Lindström Assignee: Julius Goryavsky
Resolution: Unresolved Votes: 0
Labels: None

Issue Links:
Blocks
blocks MDEV-33211 Galera SST on maria-backup causes don... In Progress

 Description   

Analysis: Here we have high-priority BF applier thread that finds a conflicting local transaction. In reproduced case it was thread updating persistent statistics. BF thread is holding lock_sys latch and thread updating persistent statistics is waiting for it. Conflicting thread is_wsrep() is false and it is holding at least one table lock and record locks. However as it is not wsrep it's trx->mysql_thd is NULL causing assertion.

How to reproduce: No simple case yet found but with 2-node cluster use sysbench and insert 20m rows. Then start sysbench measure on node_1.

(gdb) p v->mysql_thd
 
$1 = (THD *) 0x0
 
(gdb) p v->lock.wait_lock
 
$4 = {m = std::atomic<ib_lock_t *> = { 0x0 }}
 
(gdb) p v->state
 
$5 = {m = std::atomic<trx_state_t> = { TRX_STATE_ACTIVE }}
 
(gdb) p v->id
 
$6 = 39
 
(gdb) p v->wsrep
 
$7 = 0 '\000'
 
(gdb) p v->lock
 
$8 = {wait_lock = {m = std::atomic<ib_lock_t *> = { 0x0 }}, wait_trx = 0x0, cond = {__data = {__wseq = {__value64 = 0, __value32 = {__low = 0, __high = 0}}, __g1_start = {__value64 = 0, __value32 = {__low = 0, __high = 0}}, __g_refs = {0, 0}, __g_size = {0, 0}, __g1_orig_size = 0, __wrefs = 0, __g_signals = {0, 0}}, __size = '\000' <repeats 47 times>, __align = 0}, suspend_time = {m = std::atomic<my_hrtime_t> = { {val = 0} }}, was_chosen_as_deadlock_victim = {m = std::atomic<unsigned char> = { 0 '\000' }}, rec_cached = 1 '\001', table_cached = 1 '\001', wait_thr = 0x0, rec_pool = {{lock = {trx = 0x7f5b00ca7b80, trx_locks = {prev = 0x7f5b00ca8080, next = 0x0}, index = 0x55c1c4cc8de8, hash = 0x7f5b00ca9280, requested_time = 0, wait_time = 0, un_member = {tab_lock = {table = 0x900000045, locks = {prev = 0x50, next = 0x0}}, rec_lock = {page_id = {m_id = 38654705733}, n_bits = 80}}, type_mode = 1027}}, {lock = {trx = 0x0, trx_locks = {prev = 0x0, next = 0x0}, index = 0x0, hash = 0x0, requested_time = 0, wait_time = 0, un_member = {tab_lock = {table = 0x0, locks = {prev = 0x0, next = 0x0}}, rec_lock = {page_id = {m_id = 0}, n_bits = 0}}, type_mode = 0}}, {lock = {trx = 0x0, trx_locks = {prev = 0x0, next = 0x0}, index = 0x0, hash = 0x0, requested_time = 0, wait_time = 0, un_member = {tab_lock = {table = 0x0, locks = {prev = 0x0, next = 0x0}}, rec_lock = {page_id = {m_id = 0}, n_bits = 0}}, type_mode = 0}}, {lock = {trx = 0x0, trx_locks = {prev = 0x0, next = 0x0}, index = 0x0, hash = 0x0, requested_time = 0, wait_time = 0, un_member = {tab_lock = {table = 0x0, locks = {prev = 0x0, next = 0x0}}, rec_lock = {page_id = {m_id = 0}, n_bits = 0}}, type_mode = 0}}, {lock = {trx = 0x0, trx_locks = {prev = 0x0, next = 0x0}, index = 0x0, hash = 0x0, requested_time = 0, wait_time = 0, un_member = {tab_lock = {table = 0x0, locks = {prev = 0x0, next = 0x0}}, rec_lock = {page_id = {m_id = 0}, n_bits = 0}}, type_mode = 0}}, {lock = {trx = 0x0, trx_locks = {prev = 0x0, next = 0x0}, index = 0x0, hash = 0x0, requested_time = 0, wait_time = 0, un_member = {tab_lock = {table = 0x0, locks = {prev = 0x0, next = 0x0}}, rec_lock = {page_id = {m_id = 0}, n_bits = 0}}, type_mode = 0}}, {lock = {trx = 0x0, trx_locks = {prev = 0x0, next = 0x0}, index = 0x0, hash = 0x0, requested_time = 0, wait_time = 0, un_member = {tab_lock = {table = 0x0, locks = {prev = 0x0, next = 0x0}}, rec_lock = {page_id = {m_id = 0}, n_bits = 0}}, type_mode = 0}}, {lock = {trx = 0x0, trx_locks = {prev = 0x0, next = 0x0}, index = 0x0, hash = 0x0, requested_time = 0, wait_time = 0, un_member = {tab_lock = {table = 0x0, locks = {prev = 0x0, next = 0x0}}, rec_lock = {page_id = {m_id = 0}, n_bits = 0}}, type_mode = 0}}}, table_pool = {{trx = 0x7f5b00ca7b80, trx_locks = {prev = 0x0, next = 0x7f5b00ca7c80}, index = 0x0, hash = 0x0, requested_time = 0, wait_time = 0, un_member = {tab_lock = {table = 0x55c1c4cbffa8, locks = {prev = 0x0, next = 0x7f5b00ca9680}}, rec_lock = {page_id = {m_id = 94290718752680}, n_bits = 0}}, type_mode = 9}, {trx = 0x0, trx_locks = {prev = 0x0, next = 0x0}, index = 0x0, hash = 0x0, requested_time = 0, wait_time = 0, un_member = {tab_lock = {table = 0x0, locks = {prev = 0x0, next = 0x0}}, rec_lock = {page_id = {m_id = 0}, n_bits = 0}}, type_mode = 0}, {trx = 0x0, trx_locks = {prev = 0x0, next = 0x0}, index = 0x0, hash = 0x0, requested_time = 0, wait_time = 0, un_member = {tab_lock = {table = 0x0, locks = {prev = 0x0, next = 0x0}}, rec_lock = {page_id = {m_id = 0}, n_bits = 0}}, type_mode = 0}, {trx = 0x0, trx_locks = {prev = 0x0, next = 0x0}, index = 0x0, hash = 0x0, requested_time = 0, wait_time = 0, un_member = {tab_lock = {table = 0x0, locks = {prev = 0x0, next = 0x0}}, rec_lock = {page_id = {m_id = 0}, n_bits = 0}}, type_mode = 0}, {trx = 0x0, trx_locks = {prev = 0x0, next = 0x0}, index = 0x0, hash = 0x0, requested_time = 0, wait_time = 0, un_member = {tab_lock = {table = 0x0, locks = {prev = 0x0, next = 0x0}}, rec_lock = {page_id = {m_id = 0}, n_bits = 0}}, type_mode = 0}, {trx = 0x0, trx_locks = {prev = 0x0, next = 0x0}, index = 0x0, hash = 0x0, requested_time = 0, wait_time = 0, un_member = {tab_lock = {table = 0x0, locks = {prev = 0x0, next = 0x0}}, rec_lock = {page_id = {m_id = 0}, n_bits = 0}}, type_mode = 0}, {trx = 0x0, trx_locks = {prev = 0x0, next = 0x0}, index = 0x0, hash = 0x0, requested_time = 0, wait_time = 0, un_member = {tab_lock = {table = 0x0, locks = {prev = 0x0, next = 0x0}}, rec_lock = {page_id = {m_id = 0}, n_bits = 0}}, type_mode = 0}, {trx = 0x0, trx_locks = {prev = 0x0, next = 0x0}, index = 0x0, hash = 0x0, requested_time = 0, wait_time = 0, un_member = {tab_lock = {table = 0x0, locks = {prev = 0x0, next = 0x0}}, rec_lock = {page_id = {m_id = 0}, n_bits = 0}}, type_mode = 0}}, lock_heap = 0x55c1c4b80098, trx_locks = {count = 2, start = 0x7f5b00ca8080, end = 0x7f5b00ca7c80, node = &ib_lock_t::trx_locks, init = 51966}, table_locks = std::vector of length 1, capacity 1 = {0x7f5b00ca8080}, evicted_tables = {count = 0, start = 0x0, end = 0x0, node = &dict_table_t::table_LRU, init = 51966}, n_rec_locks = 1}

Stack trace (using additional assertion)

(gdb) where
 
#0  __pthread_kill_implementation (no_tid=0, signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:44
 
#1  __pthread_kill_internal (signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:78
 
#2  __GI___pthread_kill (threadid=<optimized out>, signo=6) at ./nptl/pthread_kill.c:89
 
#3  0x000055c1c1fc9781 in my_write_core (sig=6) at /home/jan/work/mariadb/upstream/mysys/stacktrace.c:424
 
#4  0x000055c1c1695719 in handle_fatal_signal (sig=6) at /home/jan/work/mariadb/upstream/sql/signal_handler.cc:357
 
#5  <signal handler called>
 
#6  __pthread_kill_implementation (no_tid=0, signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:44
 
#7  __pthread_kill_internal (signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:78
 
#8  __GI___pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
 
#9  0x00007f5b13842866 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
 
#10 0x00007f5b138268b7 in __GI_abort () at ./stdlib/abort.c:79
 
#11 0x000055c1c1d7f8a7 in ut_dbg_assertion_failed (expr=0x55c1c24630a2 "v->mysql_thd", file=0x55c1c2462470 "/home/jan/work/mariadb/upstream/storage/innobase/lock/lock0lock.cc", line=1040) at /home/jan/work/mariadb/upstream/storage/innobase/ut/ut0dbg.cc:60
 
#12 0x000055c1c1b7c29c in lock_wait_wsrep (trx=0x7f5b00ca9180) at /home/jan/work/mariadb/upstream/storage/innobase/lock/lock0lock.cc:1040
 
#13 0x000055c1c1b802a0 in lock_wait (thr=0x7f5aec0b1968) at /home/jan/work/mariadb/upstream/storage/innobase/lock/lock0lock.cc:1930
 
#14 0x000055c1c1cb44ca in row_mysql_handle_errors (new_err=0x7f5b1122bd54, trx=0x7f5b00ca9180, thr=0x7f5aec0b1968, savept=0x7f5b1122bd58) at /home/jan/work/mariadb/upstream/storage/innobase/row/row0mysql.cc:688
 
#15 0x000055c1c1cb5fcd in row_insert_for_mysql (mysql_rec=0x7f5aec0aff38 "\377\035\025", prebuilt=0x7f5aec0b10f8, ins_mode=ROW_INS_NORMAL) at /home/jan/work/mariadb/upstream/storage/innobase/row/row0mysql.cc:1326
 
#16 0x000055c1c1ac0e2b in ha_innobase::write_row (this=0x7f5aec0b03c0, record=0x7f5aec0aff38 "\377\035\025") at /home/jan/work/mariadb/upstream/storage/innobase/handler/ha_innodb.cc:7844
 
#17 0x000055c1c16b057e in handler::ha_write_row (this=0x7f5aec0b03c0, buf=0x7f5aec0aff38 "\377\035\025") at /home/jan/work/mariadb/upstream/sql/handler.cc:7661
 
#18 0x000055c1c1859443 in Rows_log_event::write_row (this=0x7f5aec02bb48, rgi=0x7f5aec021c10, overwrite=false) at /home/jan/work/mariadb/upstream/sql/log_event_server.cc:7746
 
#19 0x000055c1c1859b05 in Write_rows_log_event::do_exec_row (this=0x7f5aec02bb48, rgi=0x7f5aec021c10) at /home/jan/work/mariadb/upstream/sql/log_event_server.cc:8008
 
#20 0x000055c1c1852e75 in Rows_log_event::do_apply_event (this=0x7f5aec02bb48, rgi=0x7f5aec021c10) at /home/jan/work/mariadb/upstream/sql/log_event_server.cc:6126
 
#21 0x000055c1c1838c0e in Log_event::apply_event (this=0x7f5aec02bb48, rgi=0x7f5aec021c10) at /home/jan/work/mariadb/upstream/sql/log_event.cc:4191
 
#22 0x000055c1c1a81e8a in wsrep_apply_events (thd=0x7f5aec000dc8, rli=0x7f5aec0159a0, events_buf=0x7f5b03f085e0, buf_len=504160) at /home/jan/work/mariadb/upstream/sql/wsrep_applier.cc:213
 
#23 0x000055c1c1a59ce9 in apply_events (thd=0x7f5aec000dc8, rli=0x7f5aec0159a0, data=..., err=...) at /home/jan/work/mariadb/upstream/sql/wsrep_high_priority_service.cc:128
 
#24 0x000055c1c1a5c204 in Wsrep_applier_service::apply_write_set (this=0x7f5b1122da90, ws_meta=..., data=..., err=...) at /home/jan/work/mariadb/upstream/sql/wsrep_high_priority_service.cc:596
 
#25 0x000055c1c20ae066 in apply_write_set (server_state=..., high_priority_service=..., ws_handle=..., ws_meta=..., data=...) at /home/jan/work/mariadb/upstream/wsrep-lib/src/server_state.cpp:332
 
#26 0x000055c1c20b243c in wsrep::server_state::on_apply (this=0x55c1c4a26d70, high_priority_service=..., ws_handle=..., ws_meta=..., data=...) at /home/jan/work/mariadb/upstream/wsrep-lib/src/server_state.cpp:1128
 
#27 0x000055c1c20cd383 in wsrep::high_priority_service::apply (this=0x7f5b1122da90, ws_handle=..., ws_meta=..., data=...) at /home/jan/work/mariadb/upstream/wsrep-lib/include/wsrep/high_priority_service.hpp:47
 
#28 0x000055c1c20c9da3 in (anonymous namespace)::apply_cb (ctx=0x7f5b1122da90, wsh=0x7f5b1122ccc0, flags=65, buf=0x7f5b1122ccd0, meta=0x7f5b1122cf90, exit_loop=0x7f5b1122cf4f) at /home/jan/work/mariadb/upstream/wsrep-lib/src/wsrep_provider_v26.cpp:507
 
#29 0x00007f5b13276932 in galera::TrxHandleSlave::apply (this=this@entry=0x7f5aec06a850, recv_ctx=recv_ctx@entry=0x7f5b1122da90, apply_cb=0x55c1c20c9b67 <(anonymous namespace)::apply_cb(void*, wsrep_ws_handle_t const*, uint32_t, wsrep_buf_t const*, wsrep_trx_meta_t const*, wsrep_bool_t*)>, meta=..., exit_loop=exit_loop@entry=@0x7f5b1122cf4f: false) at /home/jan/work/galera-lib/galera/galera/src/trx_handle.cpp:396
 
#30 0x00007f5b132895b8 in galera::ReplicatorSMM::apply_trx (this=this@entry=0x55c1c4a3cd70, recv_ctx=recv_ctx@entry=0x7f5b1122da90, ts=...) at /home/jan/work/galera-lib/galera/galera/src/replicator_smm.cpp:518
 
#31 0x00007f5b1328db63 in galera::ReplicatorSMM::process_trx (this=0x55c1c4a3cd70, recv_ctx=0x7f5b1122da90, ts_ptr=...) at /home/jan/work/galera-lib/galera/galera/src/replicator_smm.cpp:2152
 
#32 0x00007f5b132c4131 in galera::GcsActionSource::process_writeset (this=0x55c1c4a6f4c0, recv_ctx=0x7f5b1122da90, act=..., exit_loop=@0x7f5b1122d67f: false) at /home/jan/work/galera-lib/galera/galera/src/gcs_action_source.cpp:62
 
#33 0x00007f5b132c5286 in galera::GcsActionSource::process (this=0x55c1c4a6f4c0, recv_ctx=0x7f5b1122da90, exit_loop=@0x7f5b1122d67f: false) at /home/jan/work/galera-lib/galera/galera/src/gcs_action_source.cpp:186
 
#34 0x00007f5b1328e2b0 in galera::ReplicatorSMM::async_recv (this=0x55c1c4a3cd70, recv_ctx=0x7f5b1122da90) at /home/jan/work/galera-lib/galera/galera/src/replicator_smm.cpp:404
 
#35 0x00007f5b13262f8f in galera_recv (gh=<optimized out>, recv_ctx=<optimized out>) at /home/jan/work/galera-lib/galera/galera/src/wsrep_provider.cpp:264
 
#36 0x000055c1c20cb45e in wsrep::wsrep_provider_v26::run_applier (this=0x55c1c4a27720, applier_ctx=0x7f5b1122da90) at /home/jan/work/mariadb/upstream/wsrep-lib/src/wsrep_provider_v26.cpp:858
 
#37 0x000055c1c1a82d9c in wsrep_replication_process (thd=0x7f5aec000dc8, arg=0x55c1c4a75440) at /home/jan/work/mariadb/upstream/sql/wsrep_thd.cc:57
 
#38 0x000055c1c1a7003b in start_wsrep_THD (arg=0x55c1c4a75440) at /home/jan/work/mariadb/upstream/sql/wsrep_mysqld.cc:3776
 
#39 0x000055c1c19d6518 in pfs_spawn_thread (arg=0x55c1c4a61688) at /home/jan/work/mariadb/upstream/storage/perfschema/pfs.cc:2201
 
#40 0x00007f5b13897ada in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:444
 
#41 0x00007f5b1392847c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78



 Comments   
Comment by Jan Lindström [ 2024-01-23 ]

https://github.com/MariaDB/server/pull/3019

Comment by Marko Mäkelä [ 2024-01-23 ]

As far as I can tell, contrary to what the pull request is currently claiming, dict_stats_save() is assigning trx_t::mysql_thd. Ever since MDEV-16678 was fixed in MariaDB Server 10.5.1, all internal InnoDB transactions should be covered metadata locks (MDL), and they are attached to a THD object. In MDEV-25506 this was extended to cover DDL operations as well.

I would be very interested to see what kind of a transaction would not be associated with a THD object. I can think of two cases: an incomplete transaction that was recovered from undo logs on server startup (and not yet rolled back), or a transaction that is in XA PREPARE state and whose client connection was disconnected (MDEV-742).

Can you please show the entire contents of the object v and the stack trace of the thread that is associated with it in the failure case?

Comment by Jan Lindström [ 2024-01-24 ]

I could reproduce using assertion but this time stack is different

#12 0x000055bf51e34cb4 in lock_wait_wsrep (trx=0x7fec23402d80) at /home/jan/work/mariadb/10.11-backup/storage/innobase/lock/lock0lock.cc:1040
 
(gdb) p v->id
 
$11 = 22537
 
(gdb) p *v
 
$12 = {<ilist_node<void>> = {next = 0x7fec23400180, prev = 0x7fec23404380}, skip_lock_inheritance_and_n_ref = {m = std::atomic<unsigned int> = { 0 }}, id = 22537, max_inactive_id = 0, mutex = {lock = std::atomic<unsigned int> = { 0 }, static HOLDER = 2147483648}, mutex_owner = std::atomic<unsigned long> = { 0 }, state = {m = std::atomic<trx_state_t> = { TRX_STATE_ACTIVE }}, lock = {wait_lock = {m = std::atomic<ib_lock_t *> = { 0x0 }}, wait_trx = 0x0, cond = {__data = {__wseq = {__value64 = 0, __value32 = {__low = 0, __high = 0}}, __g1_start = {__value64 = 0, __value32 = {__low = 0, __high = 0}}, __g_refs = {0, 0}, __g_size = {0, 0}, __g1_orig_size = 0, __wrefs = 0, __g_signals = {0, 0}}, __size = '\000' <repeats 47 times>, __align = 0}, suspend_time = {m = std::atomic<my_hrtime_t> = { {val = 0} }}, was_chosen_as_deadlock_victim = {m = std::atomic<unsigned char> = { 0 '\000' }}, rec_cached = 1 '\001', table_cached = 1 '\001', wait_thr = 0x0, rec_pool = {{lock = {trx = 0x7fec23402280, trx_locks = {prev = 0x7fec23402780, next = 0x0}, index = 0x55bf54056db8, hash = 0x7fec23402e80, requested_time = 0, wait_time = 0, un_member = {tab_lock = {table = 0x900000007, locks = {prev = 0x50, next = 0x0}}, rec_lock = {page_id = {m_id = 38654705671}, n_bits = 80}}, type_mode = 1027}}, {lock = {trx = 0x0, trx_locks = {prev = 0x0, next = 0x0}, index = 0x0, hash = 0x0, requested_time = 0, wait_time = 0, un_member = {tab_lock = {table = 0x0, locks = {prev = 0x0, next = 0x0}}, rec_lock = {page_id = {m_id = 0}, n_bits = 0}}, type_mode = 0}}, {lock = {trx = 0x0, trx_locks = {prev = 0x0, next = 0x0}, index = 0x0, hash = 0x0, requested_time = 0, wait_time = 0, un_member = {tab_lock = {table = 0x0, locks = {prev = 0x0, next = 0x0}}, rec_lock = {page_id = {m_id = 0}, n_bits = 0}}, type_mode = 0}}, {lock = {trx = 0x0, trx_locks = {prev = 0x0, next = 0x0}, index = 0x0, hash = 0x0, requested_time = 0, wait_time = 0, un_member = {tab_lock = {table = 0x0, locks = {prev = 0x0, next = 0x0}}, rec_lock = {page_id = {m_id = 0}, n_bits = 0}}, type_mode = 0}}, {lock = {trx = 0x0, trx_locks = {prev = 0x0, next = 0x0}, index = 0x0, hash = 0x0, requested_time = 0, wait_time = 0, un_member = {tab_lock = {table = 0x0, locks = {prev = 0x0, next = 0x0}}, rec_lock = {page_id = {m_id = 0}, n_bits = 0}}, type_mode = 0}}, {lock = {trx = 0x0, trx_locks = {prev = 0x0, next = 0x0}, index = 0x0, hash = 0x0, requested_time = 0, wait_time = 0, un_member = {tab_lock = {table = 0x0, locks = {prev = 0x0, next = 0x0}}, rec_lock = {page_id = {m_id = 0}, n_bits = 0}}, type_mode = 0}}, {lock = {trx = 0x0, trx_locks = {prev = 0x0, next = 0x0}, index = 0x0, hash = 0x0, requested_time = 0, wait_time = 0, un_member = {tab_lock = {table = 0x0, locks = {prev = 0x0, next = 0x0}}, rec_lock = {page_id = {m_id = 0}, n_bits = 0}}, type_mode = 0}}, {lock = {trx = 0x0, trx_locks = {prev = 0x0, next = 0x0}, index = 0x0, hash = 0x0, requested_time = 0, wait_time = 0, un_member = {tab_lock = {table = 0x0, locks = {prev = 0x0, next = 0x0}}, rec_lock = {page_id = {m_id = 0}, n_bits = 0}}, type_mode = 0}}}, table_pool = {{trx = 0x7fec23402280, trx_locks = {prev = 0x0, next = 0x7fec23402380}, index = 0x0, hash = 0x0, requested_time = 0, wait_time = 0, un_member = {tab_lock = {table = 0x55bf5404e108, locks = {prev = 0x7fec23400680, next = 0x7fec23404880}}, rec_lock = {page_id = {m_id = 94280236720392}, n_bits = 591398528}}, type_mode = 9}, {trx = 0x0, trx_locks = {prev = 0x0, next = 0x0}, index = 0x0, hash = 0x0, requested_time = 0, wait_time = 0, un_member = {tab_lock = {table = 0x0, locks = {prev = 0x0, next = 0x0}}, rec_lock = {page_id = {m_id = 0}, n_bits = 0}}, type_mode = 0}, {trx = 0x0, trx_locks = {prev = 0x0, next = 0x0}, index = 0x0, hash = 0x0, requested_time = 0, wait_time = 0, un_member = {tab_lock = {table = 0x0, locks = {prev = 0x0, next = 0x0}}, rec_lock = {page_id = {m_id = 0}, n_bits = 0}}, type_mode = 0}, {trx = 0x0, trx_locks = {prev = 0x0, next = 0x0}, index = 0x0, hash = 0x0, requested_time = 0, wait_time = 0, un_member = {tab_lock = {table = 0x0, locks = {prev = 0x0, next = 0x0}}, rec_lock = {page_id = {m_id = 0}, n_bits = 0}}, type_mode = 0}, {trx = 0x0, trx_locks = {prev = 0x0, next = 0x0}, index = 0x0, hash = 0x0, requested_time = 0, wait_time = 0, un_member = {tab_lock = {table = 0x0, locks = {prev = 0x0, next = 0x0}}, rec_lock = {page_id = {m_id = 0}, n_bits = 0}}, type_mode = 0}, {trx = 0x0, trx_locks = {prev = 0x0, next = 0x0}, index = 0x0, hash = 0x0, requested_time = 0, wait_time = 0, un_member = {tab_lock = {table = 0x0, locks = {prev = 0x0, next = 0x0}}, rec_lock = {page_id = {m_id = 0}, n_bits = 0}}, type_mode = 0}, {trx = 0x0, trx_locks = {prev = 0x0, next = 0x0}, index = 0x0, hash = 0x0, requested_time = 0, wait_time = 0, un_member = {tab_lock = {table = 0x0, locks = {prev = 0x0, next = 0x0}}, rec_lock = {page_id = {m_id = 0}, n_bits = 0}}, type_mode = 0}, {trx = 0x0, trx_locks = {prev = 0x0, next = 0x0}, index = 0x0, hash = 0x0, requested_time = 0, wait_time = 0, un_member = {tab_lock = {table = 0x0, locks = {prev = 0x0, next = 0x0}}, rec_lock = {page_id = {m_id = 0}, n_bits = 0}}, type_mode = 0}}, lock_heap = 0x55bf53f0b168, trx_locks = {count = 2, start = 0x7fec23402780, end = 0x7fec23402380, node = &ib_lock_t::trx_locks, init = 51966}, table_locks = std::vector of length 1, capacity 1 = {0x7fec23402780}, evicted_tables = {count = 0, start = 0x0, end = 0x0, node = &dict_table_t::table_LRU, init = 51966}, n_rec_locks = 1}, wsrep = 0 '\000', read_view = {<ReadViewBase> = {m_low_limit_id = 0, m_up_limit_id = 0, m_ids = std::vector of length 0, capacity 0, m_low_limit_no = 0}, m_open = std::atomic<bool> = { false }, m_mutex = {lock = std::atomic<unsigned int> = { 0 }, static HOLDER = 2147483648}, m_creator_trx_id = 0}, is_recovered = true, op_info = 0x55bf527a11c0 "", isolation_level = 2, check_foreigns = true, bulk_insert = false, is_registered = false, active_commit_ordered = false, check_unique_secondary = true, flush_log_later = false, duplicates = 0, dict_operation = false, was_dict_operation = false, dict_operation_lock_mode = false, start_time = 1706016915, start_time_micro = 607969726739, commit_lsn = 0, mysql_thd = 0x0, mysql_log_file_name = 0x0, mysql_log_offset = 0, n_mysql_tables_in_use = 0, mysql_n_tables_locked = 0, error_state = DB_SUCCESS, error_info = 0x0, error_key_num = 18446744073709551615, graph = 0x0, trx_savepoints = {count = 0, start = 0x0, end = 0x0, node = &trx_named_savept_t::trx_savepoints, init = 51966}, undo_no = 4, last_sql_stat_start = {least_undo_no = 0}, rsegs = {m_redo = {rseg = 0x55bf52ef2200 <trx_sys+27456>, undo = 0x55bf5403d2d8}, m_noredo = {rseg = 0x0, undo = 0x0}}, roll_limit = 0, in_rollback = false, pages_undone = 0, n_autoinc_rows = 0, autoinc_locks = 0x55bf5404f1c8, read_only = false, auto_commit = false, will_lock = false, apply_online_log = false, fts_trx = 0x0, fts_next_doc_id = 0, flush_tables = 0, start_line = 683, start_file = 0x55bf527a11e0 "/home/jan/work/mariadb/10.11-backup/storage/innobase/trx/trx0trx.cc", xid = {formatID = -1, gtrid_length = 94280201471197, bqual_length = 13, data = "\020\322\003T\277U\000\000@\322\003T\277U\000\000\020\322\003T\277U\000\000\220>f\a\376\177\000\000m(\337Q\277U\000\000\020\322\003T\277U\000\000\001\000\000\000\000\000\000\000@\322\003T\277U\000\000\020\322\003T\277U\000\000\260>f\a\376\177\000\000j\a\337Q\277U\000\000\r\000\000\000\000\000\000\000\020\322\003T\277U\000\000 ?f\a\376\177\000\000f\003\353Q\277U\000\000\r\000\000\000\000\000\000"}, mod_tables = std::map with 0 elements, detailed_error = 0x55bf53f0b048 "", rw_trx_hash_element = 0x55bf5404f2c8, rw_trx_hash_pins = 0x0, magic_n = 91118598}
 
(gdb) thread apply all bt
 
 
 
Thread 21 (Thread 0x7feb17fff6c0 (LWP 1334052)):
 
#0  0x00007fec4a094076 in __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x7feb17ffe960, op=137, expected=0, futex_word=0x55bf53ee5588) at ./nptl/futex-internal.c:57
 
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x7feb17ffe960, clockid=0, expected=0, futex_word=0x55bf53ee5588) at ./nptl/futex-internal.c:87
 
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x55bf53ee5588, expected=expected@entry=0, clockid=clockid@entry=1, abstime=abstime@entry=0x7feb17ffe960, private=private@entry=0) at ./nptl/futex-internal.c:139
 
#3  0x00007fec4a0972d9 in __pthread_cond_wait_common (abstime=<optimized out>, clockid=<optimized out>, mutex=0x55bf53e4a590, cond=0x55bf53ee5560) at ./nptl/pthread_cond_wait.c:503
 
#4  ___pthread_cond_clockwait64 (abstime=<optimized out>, clockid=<optimized out>, mutex=0x55bf53e4a590, cond=0x55bf53ee5560) at ./nptl/pthread_cond_wait.c:691
 
#5  ___pthread_cond_clockwait64 (cond=0x55bf53ee5560, mutex=0x55bf53e4a590, clockid=<optimized out>, abstime=<optimized out>) at ./nptl/pthread_cond_wait.c:679
 
#6  0x000055bf521daa1b in std::__condvar::wait_until (this=0x55bf53ee5560, __m=..., __clock=1, __abs_time=...) at /usr/include/c++/13/bits/std_mutex.h:185
 
#7  0x000055bf521dd047 in std::condition_variable::__wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x55bf53ee5560, __lock=..., __atime=std::chrono::_V2::steady_clock time_point = { 608029817732050ns }) at /usr/include/c++/13/condition_variable:203
 
#8  0x000055bf521dc65b in std::condition_variable::wait_until<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x55bf53ee5560, __lock=..., __atime=std::chrono::_V2::steady_clock time_point = { 608029817732050ns }) at /usr/include/c++/13/condition_variable:113
 
#9  0x000055bf521db995 in std::condition_variable::wait_for<long, std::ratio<1l, 1000l> > (this=0x55bf53ee5560, __lock=..., __rtime=std::chrono::duration = { 60000ms }) at /usr/include/c++/13/condition_variable:165
 
#10 0x000055bf521d8ea1 in tpool::thread_pool_generic::wait_for_tasks (this=0x55bf53e4a3f0, lk=..., thread_data=0x55bf53ee5560) at /home/jan/work/mariadb/10.11-backup/tpool/tpool_generic.cc:482
 
#11 0x000055bf521d90bb in tpool::thread_pool_generic::get_task (this=0x55bf53e4a3f0, thread_var=0x55bf53ee5560, t=0x7feb17ffeab0) at /home/jan/work/mariadb/10.11-backup/tpool/tpool_generic.cc:535
 
#12 0x000055bf521d941a in tpool::thread_pool_generic::worker_main (this=0x55bf53e4a3f0, thread_var=0x55bf53ee5560) at /home/jan/work/mariadb/10.11-backup/tpool/tpool_generic.cc:581
 
#13 0x000055bf521df8d0 in std::__invoke_impl<void, void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> (__f=@0x55bf54045ab8: (void (tpool::thread_pool_generic::*)(tpool::thread_pool_generic * const, tpool::worker_data *)) 0x55bf521d9312 <tpool::thread_pool_generic::worker_main(tpool::worker_data*)>, __t=@0x55bf54045ab0: 0x55bf53e4a3f0) at /usr/include/c++/13/bits/invoke.h:74
 
#14 0x000055bf521df7d6 in std::__invoke<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> (__fn=@0x55bf54045ab8: (void (tpool::thread_pool_generic::*)(tpool::thread_pool_generic * const, tpool::worker_data *)) 0x55bf521d9312 <tpool::thread_pool_generic::worker_main(tpool::worker_data*)>) at /usr/include/c++/13/bits/invoke.h:96
 
#15 0x000055bf521df6f5 in std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> >::_M_invoke<0ul, 1ul, 2ul> (this=0x55bf54045aa8) at /usr/include/c++/13/bits/std_thread.h:292
 
#16 0x000055bf521df68e in std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> >::operator() (this=0x55bf54045aa8) at /usr/include/c++/13/bits/std_thread.h:299
 
#17 0x000055bf521df66e in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> > >::_M_run (this=0x55bf54045aa0) at /usr/include/c++/13/bits/std_thread.h:244
 
#18 0x00007fec4a4e6333 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
 
#19 0x00007fec4a097ada in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:444
 
#20 0x00007fec4a12847c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
 
 
 
Thread 20 (Thread 0x7feb3dbd46c0 (LWP 1334046)):
 
#0  0x00007fec4a094076 in __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x55bf539a2ef0 <purge_worker_task+112>) at ./nptl/futex-internal.c:57
 
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x55bf539a2ef0 <purge_worker_task+112>) at ./nptl/futex-internal.c:87
 
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x55bf539a2ef0 <purge_worker_task+112>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
 
#3  0x00007fec4a0969fe in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x55bf539a2ea0 <purge_worker_task+32>, cond=0x55bf539a2ec8 <purge_worker_task+72>) at ./nptl/pthread_cond_wait.c:503
 
#4  ___pthread_cond_wait (cond=0x55bf539a2ec8 <purge_worker_task+72>, mutex=0x55bf539a2ea0 <purge_worker_task+32>) at ./nptl/pthread_cond_wait.c:627
 
#5  0x000055bf521dfff0 in tpool::waitable_task::wait (this=0x55bf539a2e80 <purge_worker_task>, lk=...) at /home/jan/work/mariadb/10.11-backup/tpool/task.cc:63
 
#6  0x000055bf521e004b in tpool::waitable_task::wait (this=0x55bf539a2e80 <purge_worker_task>) at /home/jan/work/mariadb/10.11-backup/tpool/task.cc:69
 
#7  0x000055bf51ff31ff in trx_purge_wait_for_workers_to_complete () at /home/jan/work/mariadb/10.11-backup/storage/innobase/trx/trx0purge.cc:1396
 
#8  0x000055bf51ff34fa in trx_purge (n_tasks=4, history_size=216) at /home/jan/work/mariadb/10.11-backup/storage/innobase/trx/trx0purge.cc:1490
 
#9  0x000055bf51fdd154 in purge_coordinator_state::do_purge (this=0x55bf539a3020 <purge_state>) at /home/jan/work/mariadb/10.11-backup/storage/innobase/srv/srv0srv.cc:1503
 
#10 0x000055bf51fdc35e in purge_coordinator_callback () at /home/jan/work/mariadb/10.11-backup/storage/innobase/srv/srv0srv.cc:1581
 
#11 0x000055bf521dfaee in tpool::task_group::execute (this=0x55bf539a3120 <purge_coordinator_task_group>, t=0x55bf539a31c0 <purge_coordinator_task>) at /home/jan/work/mariadb/10.11-backup/tpool/task_group.cc:70
 
#12 0x000055bf521dfe2a in tpool::task::execute (this=0x55bf539a31c0 <purge_coordinator_task>) at /home/jan/work/mariadb/10.11-backup/tpool/task.cc:32
 
#13 0x000055bf521d9403 in tpool::thread_pool_generic::worker_main (this=0x55bf53e4a3f0, thread_var=0x55bf53ee53e0) at /home/jan/work/mariadb/10.11-backup/tpool/tpool_generic.cc:583
 
#14 0x000055bf521df8d0 in std::__invoke_impl<void, void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> (__f=@0x7feb34000b88: (void (tpool::thread_pool_generic::*)(class tpool::thread_pool_generic * const, struct tpool::worker_data *)) 0x55bf521d9312 <tpool::thread_pool_generic::worker_main(tpool::worker_data*)>, __t=@0x7feb34000b80: 0x55bf53e4a3f0) at /usr/include/c++/13/bits/invoke.h:74
 
#15 0x000055bf521df7d6 in std::__invoke<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> (__fn=@0x7feb34000b88: (void (tpool::thread_pool_generic::*)(class tpool::thread_pool_generic * const, struct tpool::worker_data *)) 0x55bf521d9312 <tpool::thread_pool_generic::worker_main(tpool::worker_data*)>) at /usr/include/c++/13/bits/invoke.h:96
 
#16 0x000055bf521df6f5 in std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> >::_M_invoke<0ul, 1ul, 2ul> (this=0x7feb34000b78) at /usr/include/c++/13/bits/std_thread.h:292
 
#17 0x000055bf521df68e in std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> >::operator() (this=0x7feb34000b78) at /usr/include/c++/13/bits/std_thread.h:299
 
#18 0x000055bf521df66e in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> > >::_M_run (this=0x7feb34000b70) at /usr/include/c++/13/bits/std_thread.h:244
 
#19 0x00007fec4a4e6333 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
 
#20 0x00007fec4a097ada in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:444
 
#21 0x00007fec4a12847c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
 
 
 
Thread 19 (Thread 0x7fec4832e6c0 (LWP 1332415)):
 
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
 
#1  0x000055bf51fe9d27 in srw_mutex_impl<false>::wait (this=0x55bf54057968, lk=2147483650) at /home/jan/work/mariadb/10.11-backup/storage/innobase/sync/srw_lock.cc:244
 
#2  0x000055bf51fe8e43 in srw_mutex_impl<false>::wait_and_lock (this=0x55bf54057968) at /home/jan/work/mariadb/10.11-backup/storage/innobase/sync/srw_lock.cc:331
 
#3  0x000055bf51da4acb in srw_mutex_impl<false>::wr_lock (this=0x55bf54057968) at /home/jan/work/mariadb/10.11-backup/storage/innobase/include/srw_lock.h:132
 
#4  0x000055bf51fe981a in ssux_lock_impl<false>::rd_wait (this=0x55bf54057968) at /home/jan/work/mariadb/10.11-backup/storage/innobase/sync/srw_lock.cc:397
 
#5  0x000055bf51da459d in ssux_lock_impl<false>::rd_lock (this=0x55bf54057968) at /home/jan/work/mariadb/10.11-backup/storage/innobase/include/srw_lock.h:245
 
#6  0x000055bf51f4a894 in ssux_lock::rd_lock (this=0x55bf54057960, file=0x55bf527bbf08 "/home/jan/work/mariadb/10.11-backup/storage/innobase/btr/btr0cur.cc", line=1156) at /home/jan/work/mariadb/10.11-backup/storage/innobase/include/srw_lock.h:433
 
#7  0x000055bf51f4a9c7 in sux_lock<ssux_lock>::s_lock (this=0x55bf54057960, file=0x55bf527bbf08 "/home/jan/work/mariadb/10.11-backup/storage/innobase/btr/btr0cur.cc", line=1156) at /home/jan/work/mariadb/10.11-backup/storage/innobase/include/sux_lock.h:309
 
#8  0x000055bf51f4b2ab in mtr_t::s_lock (this=0x7fec4832bc00, file=0x55bf527bbf08 "/home/jan/work/mariadb/10.11-backup/storage/innobase/btr/btr0cur.cc", line=1156, lock=0x55bf54057960) at /home/jan/work/mariadb/10.11-backup/storage/innobase/include/mtr0mtr.h:261
 
#9  0x000055bf52070007 in btr_cur_t::search_leaf (this=0x7fec4832b9a0, tuple=0x7fec241c3e28, mode=PAGE_CUR_LE, latch_mode=BTR_MODIFY_LEAF, mtr=0x7fec4832bc00) at /home/jan/work/mariadb/10.11-backup/storage/innobase/btr/btr0cur.cc:1156
 
#10 0x000055bf51d9ff72 in btr_pcur_open (tuple=0x7fec241c3e28, mode=PAGE_CUR_LE, latch_mode=BTR_DELETE_MARK_LEAF, cursor=0x7fec4832b9a0, mtr=0x7fec4832bc00) at /home/jan/work/mariadb/10.11-backup/storage/innobase/include/btr0pcur.h:431
 
#11 0x000055bf51f9db8c in row_search_index_entry (entry=0x7fec241c3e28, mode=BTR_DELETE_MARK_LEAF, pcur=0x7fec4832b9a0, mtr=0x7fec4832bc00) at /home/jan/work/mariadb/10.11-backup/storage/innobase/row/row0row.cc:1281
 
#12 0x000055bf51fc828d in row_upd_sec_index_entry (node=0x7fec241d5d08, thr=0x7fec241d6138) at /home/jan/work/mariadb/10.11-backup/storage/innobase/row/row0upd.cc:1908
 
#13 0x000055bf51fc8e20 in row_upd_sec_step (node=0x7fec241d5d08, thr=0x7fec241d6138) at /home/jan/work/mariadb/10.11-backup/storage/innobase/row/row0upd.cc:2059
 
#14 0x000055bf51fcbd50 in row_upd (node=0x7fec241d5d08, thr=0x7fec241d6138) at /home/jan/work/mariadb/10.11-backup/storage/innobase/row/row0upd.cc:2806
 
#15 0x000055bf51fcc1e3 in row_upd_step (thr=0x7fec241d6138) at /home/jan/work/mariadb/10.11-backup/storage/innobase/row/row0upd.cc:2921
 
#16 0x000055bf51f70337 in row_update_for_mysql (prebuilt=0x7fec241d5568) at /home/jan/work/mariadb/10.11-backup/storage/innobase/row/row0mysql.cc:1682
 
#17 0x000055bf51d7bfd3 in ha_innobase::delete_row (this=0x7fec241c30f0, record=0x7fec241c2c68 "\377⼏") at /home/jan/work/mariadb/10.11-backup/storage/innobase/handler/ha_innodb.cc:8746
 
#18 0x000055bf51969837 in handler::ha_delete_row (this=0x7fec241c30f0, buf=0x7fec241c2c68 "\377⼏") at /home/jan/work/mariadb/10.11-backup/sql/handler.cc:7801
 
#19 0x000055bf51b140ae in Delete_rows_log_event::do_exec_row (this=0x7fec241b7a98, rgi=0x7fec24022fd0) at /home/jan/work/mariadb/10.11-backup/sql/log_event_server.cc:8695
 
#20 0x000055bf51b0b7a3 in Rows_log_event::do_apply_event (this=0x7fec241b7a98, rgi=0x7fec24022fd0) at /home/jan/work/mariadb/10.11-backup/sql/log_event_server.cc:6146
 
#21 0x000055bf51af12ca in Log_event::apply_event (this=0x7fec241b7a98, rgi=0x7fec24022fd0) at /home/jan/work/mariadb/10.11-backup/sql/log_event.cc:4218
 
#22 0x000055bf51d3a88e in wsrep_apply_events (thd=0x7fec24000dc8, rli=0x7fec24016d60, events_buf=0x7fec3c14e0a8, buf_len=536) at /home/jan/work/mariadb/10.11-backup/sql/wsrep_applier.cc:213
 
#23 0x000055bf51d126ed in apply_events (thd=0x7fec24000dc8, rli=0x7fec24016d60, data=..., err=...) at /home/jan/work/mariadb/10.11-backup/sql/wsrep_high_priority_service.cc:128
 
#24 0x000055bf51d14c08 in Wsrep_applier_service::apply_write_set (this=0x7fec4832da90, ws_meta=..., data=..., err=...) at /home/jan/work/mariadb/10.11-backup/sql/wsrep_high_priority_service.cc:596
 
#25 0x000055bf52366690 in apply_write_set (server_state=..., high_priority_service=..., ws_handle=..., ws_meta=..., data=...) at /home/jan/work/mariadb/10.11-backup/wsrep-lib/src/server_state.cpp:332
 
#26 0x000055bf5236aa66 in wsrep::server_state::on_apply (this=0x55bf53d97d60, high_priority_service=..., ws_handle=..., ws_meta=..., data=...) at /home/jan/work/mariadb/10.11-backup/wsrep-lib/src/server_state.cpp:1128
 
#27 0x000055bf523859ad in wsrep::high_priority_service::apply (this=0x7fec4832da90, ws_handle=..., ws_meta=..., data=...) at /home/jan/work/mariadb/10.11-backup/wsrep-lib/include/wsrep/high_priority_service.hpp:47
 
#28 0x000055bf523823cd in (anonymous namespace)::apply_cb (ctx=0x7fec4832da90, wsh=0x7fec4832ccc0, flags=65, buf=0x7fec4832ccd0, meta=0x7fec4832cf90, exit_loop=0x7fec4832cf4f) at /home/jan/work/mariadb/10.11-backup/wsrep-lib/src/wsrep_provider_v26.cpp:507
 
#29 0x00007fec49c76932 in galera::TrxHandleSlave::apply (this=this@entry=0x7fec241b74e0, recv_ctx=recv_ctx@entry=0x7fec4832da90, apply_cb=0x55bf52382191 <(anonymous namespace)::apply_cb(void*, wsrep_ws_handle_t const*, uint32_t, wsrep_buf_t const*, wsrep_trx_meta_t const*, wsrep_bool_t*)>, meta=..., exit_loop=exit_loop@entry=@0x7fec4832cf4f: false) at /home/jan/work/galera-lib/galera/galera/src/trx_handle.cpp:396
 
#30 0x00007fec49c895b8 in galera::ReplicatorSMM::apply_trx (this=this@entry=0x55bf53dadd70, recv_ctx=recv_ctx@entry=0x7fec4832da90, ts=...) at /home/jan/work/galera-lib/galera/galera/src/replicator_smm.cpp:518
 
#31 0x00007fec49c8db63 in galera::ReplicatorSMM::process_trx (this=0x55bf53dadd70, recv_ctx=0x7fec4832da90, ts_ptr=...) at /home/jan/work/galera-lib/galera/galera/src/replicator_smm.cpp:2152
 
#32 0x00007fec49cc4131 in galera::GcsActionSource::process_writeset (this=0x55bf53df0620, recv_ctx=0x7fec4832da90, act=..., exit_loop=@0x7fec4832d67f: false) at /home/jan/work/galera-lib/galera/galera/src/gcs_action_source.cpp:62
 
#33 0x00007fec49cc5286 in galera::GcsActionSource::process (this=0x55bf53df0620, recv_ctx=0x7fec4832da90, exit_loop=@0x7fec4832d67f: false) at /home/jan/work/galera-lib/galera/galera/src/gcs_action_source.cpp:186
 
#34 0x00007fec49c8e2b0 in galera::ReplicatorSMM::async_recv (this=0x55bf53dadd70, recv_ctx=0x7fec4832da90) at /home/jan/work/galera-lib/galera/galera/src/replicator_smm.cpp:404
 
#35 0x00007fec49c62f8f in galera_recv (gh=<optimized out>, recv_ctx=<optimized out>) at /home/jan/work/galera-lib/galera/galera/src/wsrep_provider.cpp:264
 
#36 0x000055bf52383a88 in wsrep::wsrep_provider_v26::run_applier (this=0x55bf53d98730, applier_ctx=0x7fec4832da90) at /home/jan/work/mariadb/10.11-backup/wsrep-lib/src/wsrep_provider_v26.cpp:858
 
#37 0x000055bf51d3b7a0 in wsrep_replication_process (thd=0x7fec24000dc8, arg=0x55bf53df65c0) at /home/jan/work/mariadb/10.11-backup/sql/wsrep_thd.cc:57
 
#38 0x000055bf51d28a3f in start_wsrep_THD (arg=0x55bf53df65c0) at /home/jan/work/mariadb/10.11-backup/sql/wsrep_mysqld.cc:3776
 
#39 0x000055bf51c8ee50 in pfs_spawn_thread (arg=0x55bf53dd2688) at /home/jan/work/mariadb/10.11-backup/storage/perfschema/pfs.cc:2201
 
#40 0x00007fec4a097ada in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:444
 
#41 0x00007fec4a12847c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
 
 
 
Thread 18 (Thread 0x7fec3bdff6c0 (LWP 1332413)):
 
#0  0x00007fec4a094076 in __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x55bf53df9634) at ./nptl/futex-internal.c:57
 
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x55bf53df9634) at ./nptl/futex-internal.c:87
 
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x55bf53df9634, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
 
#3  0x00007fec4a0969fe in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x55bf53df95c8, cond=0x55bf53df9608) at ./nptl/pthread_cond_wait.c:503
 
#4  ___pthread_cond_wait (cond=0x55bf53df9608, mutex=0x55bf53df95c8) at ./nptl/pthread_cond_wait.c:627
 
#5  0x00007fec49d07336 in gu_cond_wait_SYS (mutex=0x55bf53df95c8, cond=0x55bf53df9608) at /home/jan/work/galera-lib/galera/galerautils/src/gu_threads.h:251
 
#6  gu::Lock::wait (cond=..., this=<synthetic pointer>) at /home/jan/work/galera-lib/galera/galerautils/src/gu_lock.hpp:46
 
#7  RecvBuf::front (timeout=..., this=0x55bf53df95c8) at /home/jan/work/galera-lib/galera/gcs/src/gcs_gcomm.cpp:116
 
#8  gcomm_recv (backend=<optimized out>, msg=0x55bf53db7e48, timeout=9223372035999999999) at /home/jan/work/galera-lib/galera/gcs/src/gcs_gcomm.cpp:614
 
#9  0x00007fec49ced26f in core_msg_recv (timeout=9223372035999999999, recv_msg=0x55bf53db7e48, backend=0x55bf53db7ff0) at /home/jan/work/galera-lib/galera/gcs/src/gcs_core.cpp:484
 
#10 gcs_core_recv (conn=0x55bf53db7dd0, recv_act=recv_act@entry=0x7fec3bdfeaa0, timeout=<optimized out>) at /home/jan/work/galera-lib/galera/gcs/src/gcs_core.cpp:1205
 
#11 0x00007fec49ce798a in gcs_recv_thread (arg=0x55bf53db7ae0) at /home/jan/work/galera-lib/galera/gcs/src/gcs.cpp:1454
 
#12 0x00007fec4a097ada in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:444
 
#13 0x00007fec4a12847c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
 
 
 
Thread 17 (Thread 0x7feb3cbd26c0 (LWP 1334051)):
 
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
 
#1  0x000055bf51fe9d27 in srw_mutex_impl<false>::wait (this=0x55bf54057968, lk=2147483652) at /home/jan/work/mariadb/10.11-backup/storage/innobase/sync/srw_lock.cc:244
 
#2  0x000055bf51fe8e43 in srw_mutex_impl<false>::wait_and_lock (this=0x55bf54057968) at /home/jan/work/mariadb/10.11-backup/storage/innobase/sync/srw_lock.cc:331
 
#3  0x000055bf51da4acb in srw_mutex_impl<false>::wr_lock (this=0x55bf54057968) at /home/jan/work/mariadb/10.11-backup/storage/innobase/include/srw_lock.h:132
 
#4  0x000055bf51fe981a in ssux_lock_impl<false>::rd_wait (this=0x55bf54057968) at /home/jan/work/mariadb/10.11-backup/storage/innobase/sync/srw_lock.cc:397
 
#5  0x000055bf51da459d in ssux_lock_impl<false>::rd_lock (this=0x55bf54057968) at /home/jan/work/mariadb/10.11-backup/storage/innobase/include/srw_lock.h:245
 
#6  0x000055bf51f4a894 in ssux_lock::rd_lock (this=0x55bf54057960, file=0x55bf527bbf08 "/home/jan/work/mariadb/10.11-backup/storage/innobase/btr/btr0cur.cc", line=1156) at /home/jan/work/mariadb/10.11-backup/storage/innobase/include/srw_lock.h:433
 
#7  0x000055bf51f4a9c7 in sux_lock<ssux_lock>::s_lock (this=0x55bf54057960, file=0x55bf527bbf08 "/home/jan/work/mariadb/10.11-backup/storage/innobase/btr/btr0cur.cc", line=1156) at /home/jan/work/mariadb/10.11-backup/storage/innobase/include/sux_lock.h:309
 
#8  0x000055bf51f4b2ab in mtr_t::s_lock (this=0x7feb3cbd0c70, file=0x55bf527bbf08 "/home/jan/work/mariadb/10.11-backup/storage/innobase/btr/btr0cur.cc", line=1156, lock=0x55bf54057960) at /home/jan/work/mariadb/10.11-backup/storage/innobase/include/mtr0mtr.h:261
 
#9  0x000055bf52070007 in btr_cur_t::search_leaf (this=0x7feb3cbd09f0, tuple=0x7feb1c00a158, mode=PAGE_CUR_LE, latch_mode=BTR_MODIFY_LEAF, mtr=0x7feb3cbd0c70) at /home/jan/work/mariadb/10.11-backup/storage/innobase/btr/btr0cur.cc:1156
 
#10 0x000055bf51d9ff72 in btr_pcur_open (tuple=0x7feb1c00a158, mode=PAGE_CUR_LE, latch_mode=BTR_PURGE_LEAF, cursor=0x7feb3cbd09f0, mtr=0x7feb3cbd0c70) at /home/jan/work/mariadb/10.11-backup/storage/innobase/include/btr0pcur.h:431
 
#11 0x000055bf51f9db8c in row_search_index_entry (entry=0x7feb1c00a158, mode=BTR_PURGE_LEAF, pcur=0x7feb3cbd09f0, mtr=0x7feb3cbd0c70) at /home/jan/work/mariadb/10.11-backup/storage/innobase/row/row0row.cc:1281
 
#12 0x000055bf51f91123 in row_purge_remove_sec_if_poss_leaf (node=0x55bf540221e0, index=0x55bf540577e8, entry=0x7feb1c00a158) at /home/jan/work/mariadb/10.11-backup/storage/innobase/row/row0purge.cc:474
 
#13 0x000055bf51f9163d in row_purge_remove_sec_if_poss (node=0x55bf540221e0, index=0x55bf540577e8, entry=0x7feb1c00a158) at /home/jan/work/mariadb/10.11-backup/storage/innobase/row/row0purge.cc:575
 
#14 0x000055bf51f92373 in row_purge_upd_exist_or_extern_func (thr=0x55bf54022140, node=0x55bf540221e0, undo_rec=0x7febfd0f0110 "\0019\f") at /home/jan/work/mariadb/10.11-backup/storage/innobase/row/row0purge.cc:755
 
#15 0x000055bf51f93b01 in row_purge_record_func (node=0x55bf540221e0, undo_rec=0x7febfd0f0110 "\0019\f", thr=0x55bf54022140, updated_extern=false) at /home/jan/work/mariadb/10.11-backup/storage/innobase/row/row0purge.cc:1146
 
#16 0x000055bf51f93c4c in row_purge (node=0x55bf540221e0, undo_rec=0x7febfd0f0110 "\0019\f", thr=0x55bf54022140) at /home/jan/work/mariadb/10.11-backup/storage/innobase/row/row0purge.cc:1185
 
#17 0x000055bf51f93d2b in row_purge_step (thr=0x55bf54022140) at /home/jan/work/mariadb/10.11-backup/storage/innobase/row/row0purge.cc:1248
 
#18 0x000055bf51efe43b in que_thr_step (thr=0x55bf54022140) at /home/jan/work/mariadb/10.11-backup/storage/innobase/que/que0que.cc:588
 
#19 0x000055bf51efe6dc in que_run_threads_low (thr=0x55bf54022140) at /home/jan/work/mariadb/10.11-backup/storage/innobase/que/que0que.cc:644
 
#20 0x000055bf51efe812 in que_run_threads (thr=0x55bf54022140) at /home/jan/work/mariadb/10.11-backup/storage/innobase/que/que0que.cc:664
 
#21 0x000055bf51fdbcb0 in srv_task_execute () at /home/jan/work/mariadb/10.11-backup/storage/innobase/srv/srv0srv.cc:1437
 
#22 0x000055bf51fdc311 in purge_worker_callback () at /home/jan/work/mariadb/10.11-backup/storage/innobase/srv/srv0srv.cc:1572
 
#23 0x000055bf521dfaee in tpool::task_group::execute (this=0x55bf539a3080 <purge_task_group>, t=0x55bf539a2e80 <purge_worker_task>) at /home/jan/work/mariadb/10.11-backup/tpool/task_group.cc:70
 
#24 0x000055bf521dfe2a in tpool::task::execute (this=0x55bf539a2e80 <purge_worker_task>) at /home/jan/work/mariadb/10.11-backup/tpool/task.cc:32
 
#25 0x000055bf521d9403 in tpool::thread_pool_generic::worker_main (this=0x55bf53e4a3f0, thread_var=0x55bf53ee54e0) at /home/jan/work/mariadb/10.11-backup/tpool/tpool_generic.cc:583
 
#26 0x000055bf521df8d0 in std::__invoke_impl<void, void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> (__f=@0x7feb34000d48: (void (tpool::thread_pool_generic::*)(class tpool::thread_pool_generic * const, struct tpool::worker_data *)) 0x55bf521d9312 <tpool::thread_pool_generic::worker_main(tpool::worker_data*)>, __t=@0x7feb34000d40: 0x55bf53e4a3f0) at /usr/include/c++/13/bits/invoke.h:74
 
#27 0x000055bf521df7d6 in std::__invoke<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> (__fn=@0x7feb34000d48: (void (tpool::thread_pool_generic::*)(class tpool::thread_pool_generic * const, struct tpool::worker_data *)) 0x55bf521d9312 <tpool::thread_pool_generic::worker_main(tpool::worker_data*)>) at /usr/include/c++/13/bits/invoke.h:96
 
#28 0x000055bf521df6f5 in std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> >::_M_invoke<0ul, 1ul, 2ul> (this=0x7feb34000d38) at /usr/include/c++/13/bits/std_thread.h:292
 
#29 0x000055bf521df68e in std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> >::operator() (this=0x7feb34000d38) at /usr/include/c++/13/bits/std_thread.h:299
 
#30 0x000055bf521df66e in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> > >::_M_run (this=0x7feb34000d30) at /usr/include/c++/13/bits/std_thread.h:244
 
#31 0x00007fec4a4e6333 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
 
#32 0x00007fec4a097ada in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:444
 
#33 0x00007fec4a12847c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
 
 
 
Thread 16 (Thread 0x7fec4807c6c0 (LWP 1334048)):
 
#0  0x00007fec4a094076 in __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x55bf530f15ac <COND_manager+44>) at ./nptl/futex-internal.c:57
 
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x55bf530f15ac <COND_manager+44>) at ./nptl/futex-internal.c:87
 
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x55bf530f15ac <COND_manager+44>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
 
#3  0x00007fec4a0969fe in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x55bf530f14e8 <LOCK_manager+40>, cond=0x55bf530f1580 <COND_manager>) at ./nptl/pthread_cond_wait.c:503
 
#4  ___pthread_cond_wait (cond=0x55bf530f1580 <COND_manager>, mutex=0x55bf530f14e8 <LOCK_manager+40>) at ./nptl/pthread_cond_wait.c:627
 
#5  0x000055bf52287126 in safe_cond_wait (cond=0x55bf530f1580 <COND_manager>, mp=0x55bf530f14c0 <LOCK_manager>, file=0x55bf523e1e00 "/home/jan/work/mariadb/10.11-backup/include/mysql/psi/mysql_thread.h", line=1072) at /home/jan/work/mariadb/10.11-backup/mysys/thr_mutex.c:494
 
#6  0x000055bf51545554 in inline_mysql_cond_wait (that=0x55bf530f1580 <COND_manager>, mutex=0x55bf530f14c0 <LOCK_manager>, src_file=0x55bf523e2480 "/home/jan/work/mariadb/10.11-backup/sql/sql_manager.cc", src_line=103) at /home/jan/work/mariadb/10.11-backup/include/mysql/psi/mysql_thread.h:1072
 
#7  0x000055bf51545974 in handle_manager (arg=0x0) at /home/jan/work/mariadb/10.11-backup/sql/sql_manager.cc:103
 
#8  0x000055bf51c8ee50 in pfs_spawn_thread (arg=0x55bf542a7358) at /home/jan/work/mariadb/10.11-backup/storage/perfschema/pfs.cc:2201
 
#9  0x00007fec4a097ada in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:444
 
#10 0x00007fec4a12847c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
 
 
 
Thread 15 (Thread 0x7fec233fd6c0 (LWP 1334043)):
 
#0  0x00007fec4b0ad8d7 in ?? () from /lib/x86_64-linux-gnu/liburing.so.2
 
#1  0x00007fec4b0ada2d in __io_uring_get_cqe () from /lib/x86_64-linux-gnu/liburing.so.2
 
#2  0x000055bf521e0548 in io_uring_wait_cqe_nr (ring=0x55bf53f27798, cqe_ptr=0x7fec233fcad0, wait_nr=1) at /usr/include/liburing.h:1237
 
#3  0x000055bf521e06de in io_uring_wait_cqe (ring=0x55bf53f27798, cqe_ptr=0x7fec233fcad0) at /usr/include/liburing.h:1310
 
#4  0x000055bf521e0f88 in (anonymous namespace)::aio_uring::thread_routine (aio=0x55bf53f27790) at /home/jan/work/mariadb/10.11-backup/tpool/aio_liburing.cc:145
 
#5  0x000055bf521e16c8 in std::__invoke_impl<void, void (*)((anonymous namespace)::aio_uring*), (anonymous namespace)::aio_uring*> (__f=@0x55bf53e13a60: 0x55bf521e0f61 <(anonymous namespace)::aio_uring::thread_routine((anonymous namespace)::aio_uring*)>) at /usr/include/c++/13/bits/invoke.h:61
 
#6  0x000055bf521e164b in std::__invoke<void (*)((anonymous namespace)::aio_uring*), (anonymous namespace)::aio_uring*> (__fn=@0x55bf53e13a60: 0x55bf521e0f61 <(anonymous namespace)::aio_uring::thread_routine((anonymous namespace)::aio_uring*)>) at /usr/include/c++/13/bits/invoke.h:96
 
#7  0x000055bf521e15bb in std::thread::_Invoker<std::tuple<void (*)((anonymous namespace)::aio_uring*), (anonymous namespace)::aio_uring*> >::_M_invoke<0, 1> (this=0x55bf53e13a58) at /usr/include/c++/13/bits/std_thread.h:292
 
#8  0x000055bf521e1574 in std::thread::_Invoker<std::tuple<void (*)((anonymous namespace)::aio_uring*), (anonymous namespace)::aio_uring*> >::operator() (this=0x55bf53e13a58) at /usr/include/c++/13/bits/std_thread.h:299
 
#9  0x000055bf521e1558 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (*)((anonymous namespace)::aio_uring*), (anonymous namespace)::aio_uring*> > >::_M_run (this=0x55bf53e13a50) at /usr/include/c++/13/bits/std_thread.h:244
 
#10 0x00007fec4a4e6333 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
 
#11 0x00007fec4a097ada in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:444
 
#12 0x00007fec4a12847c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
 
 
 
Thread 14 (Thread 0x7feb3d3d36c0 (LWP 1334047)):
 
#0  0x00007fec4a094076 in __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x7feb3d3d2960, op=137, expected=0, futex_word=0x55bf53ee548c) at ./nptl/futex-internal.c:57
 
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x7feb3d3d2960, clockid=32747, expected=0, futex_word=0x55bf53ee548c) at ./nptl/futex-internal.c:87
 
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x55bf53ee548c, expected=expected@entry=0, clockid=clockid@entry=1, abstime=abstime@entry=0x7feb3d3d2960, private=private@entry=0) at ./nptl/futex-internal.c:139
 
#3  0x00007fec4a0972d9 in __pthread_cond_wait_common (abstime=<optimized out>, clockid=<optimized out>, mutex=0x55bf53e4a590, cond=0x55bf53ee5460) at ./nptl/pthread_cond_wait.c:503
 
#4  ___pthread_cond_clockwait64 (abstime=<optimized out>, clockid=<optimized out>, mutex=0x55bf53e4a590, cond=0x55bf53ee5460) at ./nptl/pthread_cond_wait.c:691
 
#5  ___pthread_cond_clockwait64 (cond=0x55bf53ee5460, mutex=0x55bf53e4a590, clockid=<optimized out>, abstime=<optimized out>) at ./nptl/pthread_cond_wait.c:679
 
#6  0x000055bf521daa1b in std::__condvar::wait_until (this=0x55bf53ee5460, __m=..., __clock=1, __abs_time=...) at /usr/include/c++/13/bits/std_mutex.h:185
 
#7  0x000055bf521dd047 in std::condition_variable::__wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x55bf53ee5460, __lock=..., __atime=std::chrono::_V2::steady_clock time_point = { 608035749029396ns }) at /usr/include/c++/13/condition_variable:203
 
#8  0x000055bf521dc65b in std::condition_variable::wait_until<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x55bf53ee5460, __lock=..., __atime=std::chrono::_V2::steady_clock time_point = { 608035749029396ns }) at /usr/include/c++/13/condition_variable:113
 
#9  0x000055bf521db995 in std::condition_variable::wait_for<long, std::ratio<1l, 1000l> > (this=0x55bf53ee5460, __lock=..., __rtime=std::chrono::duration = { 60000ms }) at /usr/include/c++/13/condition_variable:165
 
#10 0x000055bf521d8ea1 in tpool::thread_pool_generic::wait_for_tasks (this=0x55bf53e4a3f0, lk=..., thread_data=0x55bf53ee5460) at /home/jan/work/mariadb/10.11-backup/tpool/tpool_generic.cc:482
 
#11 0x000055bf521d90bb in tpool::thread_pool_generic::get_task (this=0x55bf53e4a3f0, thread_var=0x55bf53ee5460, t=0x7feb3d3d2ab0) at /home/jan/work/mariadb/10.11-backup/tpool/tpool_generic.cc:535
 
#12 0x000055bf521d941a in tpool::thread_pool_generic::worker_main (this=0x55bf53e4a3f0, thread_var=0x55bf53ee5460) at /home/jan/work/mariadb/10.11-backup/tpool/tpool_generic.cc:581
 
#13 0x000055bf521df8d0 in std::__invoke_impl<void, void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> (__f=@0x55bf54057f18: (void (tpool::thread_pool_generic::*)(class tpool::thread_pool_generic * const, struct tpool::worker_data *)) 0x55bf521d9312 <tpool::thread_pool_generic::worker_main(tpool::worker_data*)>, __t=@0x55bf54057f10: 0x55bf53e4a3f0) at /usr/include/c++/13/bits/invoke.h:74
 
#14 0x000055bf521df7d6 in std::__invoke<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> (__fn=@0x55bf54057f18: (void (tpool::thread_pool_generic::*)(class tpool::thread_pool_generic * const, struct tpool::worker_data *)) 0x55bf521d9312 <tpool::thread_pool_generic::worker_main(tpool::worker_data*)>) at /usr/include/c++/13/bits/invoke.h:96
 
#15 0x000055bf521df6f5 in std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> >::_M_invoke<0ul, 1ul, 2ul> (this=0x55bf54057f08) at /usr/include/c++/13/bits/std_thread.h:292
 
#16 0x000055bf521df68e in std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> >::operator() (this=0x55bf54057f08) at /usr/include/c++/13/bits/std_thread.h:299
 
#17 0x000055bf521df66e in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> > >::_M_run (this=0x55bf54057f00) at /usr/include/c++/13/bits/std_thread.h:244
 
#18 0x00007fec4a4e6333 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
 
#19 0x00007fec4a097ada in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:444
 
#20 0x00007fec4a12847c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
 
 
 
Thread 13 (Thread 0x7fec49afe6c0 (LWP 1332408)):
 
#0  0x00007fec4a094076 in __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x55bf53dae6c4) at ./nptl/futex-internal.c:57
 
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x55bf53dae6c4) at ./nptl/futex-internal.c:87
 
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x55bf53dae6c4, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
 
#3  0x00007fec4a0969fe in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x55bf53dae658, cond=0x55bf53dae698) at ./nptl/pthread_cond_wait.c:503
 
#4  ___pthread_cond_wait (cond=0x55bf53dae698, mutex=0x55bf53dae658) at ./nptl/pthread_cond_wait.c:627
 
#5  0x00007fec49cbcace in gu_cond_wait_SYS (mutex=0x55bf53dae658, cond=0x55bf53dae698) at /home/jan/work/galera-lib/galera/galerautils/src/gu_threads.h:251
 
#6  gu::Lock::wait (cond=..., this=<synthetic pointer>) at /home/jan/work/galera-lib/galera/galerautils/src/gu_lock.hpp:46
 
#7  galera::ServiceThd::thd_func (arg=0x55bf53dae638) at /home/jan/work/galera-lib/galera/galera/src/galera_service_thd.cpp:31
 
#8  0x00007fec4a097ada in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:444
 
#9  0x00007fec4a12847c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
 
 
 
Thread 12 (Thread 0x7fec20b7c6c0 (LWP 1334044)):
 
#0  0x00007fec4a094076 in __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x55bf52efc138 <buf_pool+17464>) at ./nptl/futex-internal.c:57
 
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x55bf52efc138 <buf_pool+17464>) at ./nptl/futex-internal.c:87
 
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x55bf52efc138 <buf_pool+17464>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
 
#3  0x00007fec4a0969fe in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x55bf52efc028 <buf_pool+17192>, cond=0x55bf52efc110 <buf_pool+17424>) at ./nptl/pthread_cond_wait.c:503
 
#4  ___pthread_cond_wait (cond=0x55bf52efc110 <buf_pool+17424>, mutex=0x55bf52efc028 <buf_pool+17192>) at ./nptl/pthread_cond_wait.c:627
 
#5  0x000055bf52287126 in safe_cond_wait (cond=0x55bf52efc110 <buf_pool+17424>, mp=0x55bf52efc000 <buf_pool+17152>, file=0x55bf527d61d8 "/home/jan/work/mariadb/10.11-backup/storage/innobase/buf/buf0flu.cc", line=2441) at /home/jan/work/mariadb/10.11-backup/mysys/thr_mutex.c:494
 
#6  0x000055bf520d0593 in buf_flush_page_cleaner () at /home/jan/work/mariadb/10.11-backup/storage/innobase/buf/buf0flu.cc:2441
 
#7  0x000055bf520d4f1b in std::__invoke_impl<void, void (*)()> (__f=@0x55bf53e14298: 0x55bf520d01ec <buf_flush_page_cleaner()>) at /usr/include/c++/13/bits/invoke.h:61
 
#8  0x000055bf520d4ec7 in std::__invoke<void (*)()> (__fn=@0x55bf53e14298: 0x55bf520d01ec <buf_flush_page_cleaner()>) at /usr/include/c++/13/bits/invoke.h:96
 
#9  0x000055bf520d4e68 in std::thread::_Invoker<std::tuple<void (*)()> >::_M_invoke<0ul> (this=0x55bf53e14298) at /usr/include/c++/13/bits/std_thread.h:292
 
#10 0x000055bf520d4e38 in std::thread::_Invoker<std::tuple<void (*)()> >::operator() (this=0x55bf53e14298) at /usr/include/c++/13/bits/std_thread.h:299
 
#11 0x000055bf520d4e18 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (*)()> > >::_M_run (this=0x55bf53e14290) at /usr/include/c++/13/bits/std_thread.h:244
 
#12 0x00007fec4a4e6333 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
 
#13 0x00007fec4a097ada in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:444
 
#14 0x00007fec4a12847c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
 
 
 
Thread 11 (Thread 0x7fec43fff6c0 (LWP 1334049)):
 
#0  0x00007fec4a094076 in __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x55bf531234c8 <mysql_bin_log+4296>) at ./nptl/futex-internal.c:57
 
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x55bf531234c8 <mysql_bin_log+4296>) at ./nptl/futex-internal.c:87
 
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x55bf531234c8 <mysql_bin_log+4296>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
 
#3  0x00007fec4a0969fe in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x55bf53123418 <mysql_bin_log+4120>, cond=0x55bf531234a0 <mysql_bin_log+4256>) at ./nptl/pthread_cond_wait.c:503
 
#4  ___pthread_cond_wait (cond=0x55bf531234a0 <mysql_bin_log+4256>, mutex=0x55bf53123418 <mysql_bin_log+4120>) at ./nptl/pthread_cond_wait.c:627
 
#5  0x000055bf52287126 in safe_cond_wait (cond=0x55bf531234a0 <mysql_bin_log+4256>, mp=0x55bf531233f0 <mysql_bin_log+4080>, file=0x55bf5260b8f0 "/home/jan/work/mariadb/10.11-backup/include/mysql/psi/mysql_thread.h", line=1072) at /home/jan/work/mariadb/10.11-backup/mysys/thr_mutex.c:494
 
#6  0x000055bf51ab80b6 in inline_mysql_cond_wait (that=0x55bf531234a0 <mysql_bin_log+4256>, mutex=0x55bf531233f0 <mysql_bin_log+4080>, src_file=0x55bf5260cf98 "/home/jan/work/mariadb/10.11-backup/sql/log.cc", src_line=10847) at /home/jan/work/mariadb/10.11-backup/include/mysql/psi/mysql_thread.h:1072
 
#7  0x000055bf51adacfa in binlog_background_thread (arg=0x0) at /home/jan/work/mariadb/10.11-backup/sql/log.cc:10847
 
#8  0x000055bf51c8ee50 in pfs_spawn_thread (arg=0x55bf53e20e48) at /home/jan/work/mariadb/10.11-backup/storage/perfschema/pfs.cc:2201
 
#9  0x00007fec4a097ada in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:444
 
#10 0x00007fec4a12847c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
 
 
 
Thread 10 (Thread 0x7fec48b7a6c0 (LWP 1332409)):
 
#0  0x00007fec4a1288a6 in epoll_wait (epfd=6, events=0x7fec48b79180, maxevents=128, timeout=-1) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30
 
#1  0x00007fec49e2884e in asio::detail::epoll_reactor::run (this=0x55bf53d897e0, usec=<optimized out>, ops=...) at /home/jan/work/galera-lib/galera/asio/asio/detail/impl/epoll_reactor.ipp:470
 
#2  0x00007fec49e1feb9 in asio::detail::scheduler::do_run_one (ec=..., this_thread=..., lock=..., this=0x55bf53df43a0) at /home/jan/work/galera-lib/galera/asio/asio/detail/impl/scheduler.ipp:430
 
#3  asio::detail::scheduler::run (this=0x55bf53df43a0, ec=std::error_code = {std::_V2::error_category: 0}) at /home/jan/work/galera-lib/galera/asio/asio/detail/impl/scheduler.ipp:199
 
#4  0x00007fec49e22a0e in asio::io_context::run (this=<optimized out>) at /home/jan/work/galera-lib/galera/asio/asio/impl/io_context.ipp:62
 
#5  gu::AsioIoService::run (this=<optimized out>) at /home/jan/work/galera-lib/galera/galerautils/src/gu_asio.cpp:760
 
#6  0x00007fec49d064dd in GCommConn::run (this=0x55bf53df93b0) at /home/jan/work/galera-lib/galera/gcs/src/gcs_gcomm.cpp:439
 
#7  0x00007fec49d06d2a in run_fn (arg=<optimized out>) at /home/jan/work/galera-lib/galera/gcs/src/gcs_gcomm.cpp:307
 
#8  0x00007fec4a097ada in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:444
 
#9  0x00007fec4a12847c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
 
 
 
Thread 9 (Thread 0x7fec483796c0 (LWP 1332414)):
 
#0  0x00007fec4a094076 in __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7fec30016dc0) at ./nptl/futex-internal.c:57
 
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x7fec30016dc0) at ./nptl/futex-internal.c:87
 
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7fec30016dc0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
 
#3  0x00007fec4a0969fe in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7fec30016d10, cond=0x7fec30016d98) at ./nptl/pthread_cond_wait.c:503
 
#4  ___pthread_cond_wait (cond=0x7fec30016d98, mutex=0x7fec30016d10) at ./nptl/pthread_cond_wait.c:627
 
#5  0x000055bf52287126 in safe_cond_wait (cond=0x7fec30016d98, mp=0x7fec30016ce8, file=0x55bf526e2ff0 "/home/jan/work/mariadb/10.11-backup/include/mysql/psi/mysql_thread.h", line=1072) at /home/jan/work/mariadb/10.11-backup/mysys/thr_mutex.c:494
 
#6  0x000055bf51d3b37d in inline_mysql_cond_wait (that=0x7fec30016d98, mutex=0x7fec30016ce8, src_file=0x55bf526e38b0 "/home/jan/work/mariadb/10.11-backup/sql/wsrep_thd.h", src_line=69) at /home/jan/work/mariadb/10.11-backup/include/mysql/psi/mysql_thread.h:1072
 
#7  0x000055bf51d3ed16 in Wsrep_thd_queue::pop_front (this=0x7fec30016c90) at /home/jan/work/mariadb/10.11-backup/sql/wsrep_thd.h:69
 
#8  0x000055bf51d3c7e2 in wsrep_rollback_process (rollbacker=0x7fec30000dc8, arg=0x55bf53df65a0) at /home/jan/work/mariadb/10.11-backup/sql/wsrep_thd.cc:236
 
#9  0x000055bf51d28a3f in start_wsrep_THD (arg=0x55bf53df65a0) at /home/jan/work/mariadb/10.11-backup/sql/wsrep_mysqld.cc:3776
 
#10 0x000055bf51c8ee50 in pfs_spawn_thread (arg=0x55bf53dd07c8) at /home/jan/work/mariadb/10.11-backup/storage/perfschema/pfs.cc:2201
 
#11 0x00007fec4a097ada in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:444
 
#12 0x00007fec4a12847c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
 
 
 
Thread 8 (Thread 0x7fec43f696c0 (LWP 1334053)):
 
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
 
#1  0x000055bf51fe9d27 in srw_mutex_impl<false>::wait (this=0x55bf54057968, lk=2147483651) at /home/jan/work/mariadb/10.11-backup/storage/innobase/sync/srw_lock.cc:244
 
#2  0x000055bf51fe8e43 in srw_mutex_impl<false>::wait_and_lock (this=0x55bf54057968) at /home/jan/work/mariadb/10.11-backup/storage/innobase/sync/srw_lock.cc:331
 
#3  0x000055bf51da4acb in srw_mutex_impl<false>::wr_lock (this=0x55bf54057968) at /home/jan/work/mariadb/10.11-backup/storage/innobase/include/srw_lock.h:132
 
#4  0x000055bf51fe981a in ssux_lock_impl<false>::rd_wait (this=0x55bf54057968) at /home/jan/work/mariadb/10.11-backup/storage/innobase/sync/srw_lock.cc:397
 
#5  0x000055bf51da459d in ssux_lock_impl<false>::rd_lock (this=0x55bf54057968) at /home/jan/work/mariadb/10.11-backup/storage/innobase/include/srw_lock.h:245
 
#6  0x000055bf51f4a894 in ssux_lock::rd_lock (this=0x55bf54057960, file=0x55bf527bbf08 "/home/jan/work/mariadb/10.11-backup/storage/innobase/btr/btr0cur.cc", line=1156) at /home/jan/work/mariadb/10.11-backup/storage/innobase/include/srw_lock.h:433
 
#7  0x000055bf51f4a9c7 in sux_lock<ssux_lock>::s_lock (this=0x55bf54057960, file=0x55bf527bbf08 "/home/jan/work/mariadb/10.11-backup/storage/innobase/btr/btr0cur.cc", line=1156) at /home/jan/work/mariadb/10.11-backup/storage/innobase/include/sux_lock.h:309
 
#8  0x000055bf51f4b2ab in mtr_t::s_lock (this=0x7fec43f66bb0, file=0x55bf527bbf08 "/home/jan/work/mariadb/10.11-backup/storage/innobase/btr/btr0cur.cc", line=1156, lock=0x55bf54057960) at /home/jan/work/mariadb/10.11-backup/storage/innobase/include/mtr0mtr.h:261
 
#9  0x000055bf52070007 in btr_cur_t::search_leaf (this=0x7fec43f66950, tuple=0x7feb08025b68, mode=PAGE_CUR_LE, latch_mode=BTR_MODIFY_LEAF, mtr=0x7fec43f66bb0) at /home/jan/work/mariadb/10.11-backup/storage/innobase/btr/btr0cur.cc:1156
 
#10 0x000055bf51d9ff72 in btr_pcur_open (tuple=0x7feb08025b68, mode=PAGE_CUR_LE, latch_mode=BTR_DELETE_MARK_LEAF, cursor=0x7fec43f66950, mtr=0x7fec43f66bb0) at /home/jan/work/mariadb/10.11-backup/storage/innobase/include/btr0pcur.h:431
 
#11 0x000055bf51f9db8c in row_search_index_entry (entry=0x7feb08025b68, mode=BTR_DELETE_MARK_LEAF, pcur=0x7fec43f66950, mtr=0x7fec43f66bb0) at /home/jan/work/mariadb/10.11-backup/storage/innobase/row/row0row.cc:1281
 
#12 0x000055bf51fc828d in row_upd_sec_index_entry (node=0x7fec241be318, thr=0x7fec241be748) at /home/jan/work/mariadb/10.11-backup/storage/innobase/row/row0upd.cc:1908
 
#13 0x000055bf51fc8e20 in row_upd_sec_step (node=0x7fec241be318, thr=0x7fec241be748) at /home/jan/work/mariadb/10.11-backup/storage/innobase/row/row0upd.cc:2059
 
#14 0x000055bf51fcbd50 in row_upd (node=0x7fec241be318, thr=0x7fec241be748) at /home/jan/work/mariadb/10.11-backup/storage/innobase/row/row0upd.cc:2806
 
#15 0x000055bf51fcc1e3 in row_upd_step (thr=0x7fec241be748) at /home/jan/work/mariadb/10.11-backup/storage/innobase/row/row0upd.cc:2921
 
#16 0x000055bf51f70337 in row_update_for_mysql (prebuilt=0x7fec241bdb78) at /home/jan/work/mariadb/10.11-backup/storage/innobase/row/row0mysql.cc:1682
 
#17 0x000055bf51d7b9f9 in ha_innobase::update_row (this=0x7fec241bce40, old_row=0x7fec241bca78 "\377>\003", new_row=0x7fec241bc9b8 "\377>\003") at /home/jan/work/mariadb/10.11-backup/storage/innobase/handler/ha_innodb.cc:8625
 
#18 0x000055bf5196903d in handler::ha_update_row (this=0x7fec241bce40, old_data=0x7fec241bca78 "\377>\003", new_data=0x7fec241bc9b8 "\377>\003") at /home/jan/work/mariadb/10.11-backup/sql/handler.cc:7722
 
#19 0x000055bf51b14e56 in Update_rows_log_event::do_exec_row (this=0x7feb08022be8, rgi=0x7feb08021950) at /home/jan/work/mariadb/10.11-backup/sql/log_event_server.cc:8922
 
#20 0x000055bf51b0b7a3 in Rows_log_event::do_apply_event (this=0x7feb08022be8, rgi=0x7feb08021950) at /home/jan/work/mariadb/10.11-backup/sql/log_event_server.cc:6146
 
#21 0x000055bf51af12ca in Log_event::apply_event (this=0x7feb08022be8, rgi=0x7feb08021950) at /home/jan/work/mariadb/10.11-backup/sql/log_event.cc:4218
 
#22 0x000055bf51d3a88e in wsrep_apply_events (thd=0x7feb08000dc8, rli=0x7feb08015a40, events_buf=0x7fec3c14e940, buf_len=1497) at /home/jan/work/mariadb/10.11-backup/sql/wsrep_applier.cc:213
 
#23 0x000055bf51d126ed in apply_events (thd=0x7feb08000dc8, rli=0x7feb08015a40, data=..., err=...) at /home/jan/work/mariadb/10.11-backup/sql/wsrep_high_priority_service.cc:128
 
#24 0x000055bf51d14c08 in Wsrep_applier_service::apply_write_set (this=0x7fec43f68a90, ws_meta=..., data=..., err=...) at /home/jan/work/mariadb/10.11-backup/sql/wsrep_high_priority_service.cc:596
 
#25 0x000055bf52366690 in apply_write_set (server_state=..., high_priority_service=..., ws_handle=..., ws_meta=..., data=...) at /home/jan/work/mariadb/10.11-backup/wsrep-lib/src/server_state.cpp:332
 
#26 0x000055bf5236aa66 in wsrep::server_state::on_apply (this=0x55bf53d97d60, high_priority_service=..., ws_handle=..., ws_meta=..., data=...) at /home/jan/work/mariadb/10.11-backup/wsrep-lib/src/server_state.cpp:1128
 
#27 0x000055bf523859ad in wsrep::high_priority_service::apply (this=0x7fec43f68a90, ws_handle=..., ws_meta=..., data=...) at /home/jan/work/mariadb/10.11-backup/wsrep-lib/include/wsrep/high_priority_service.hpp:47
 
#28 0x000055bf523823cd in (anonymous namespace)::apply_cb (ctx=0x7fec43f68a90, wsh=0x7fec43f67cc0, flags=65, buf=0x7fec43f67cd0, meta=0x7fec43f67f90, exit_loop=0x7fec43f67f4f) at /home/jan/work/mariadb/10.11-backup/wsrep-lib/src/wsrep_provider_v26.cpp:507
 
#29 0x00007fec49c76932 in galera::TrxHandleSlave::apply (this=this@entry=0x7feb08022600, recv_ctx=recv_ctx@entry=0x7fec43f68a90, apply_cb=0x55bf52382191 <(anonymous namespace)::apply_cb(void*, wsrep_ws_handle_t const*, uint32_t, wsrep_buf_t const*, wsrep_trx_meta_t const*, wsrep_bool_t*)>, meta=..., exit_loop=exit_loop@entry=@0x7fec43f67f4f: false) at /home/jan/work/galera-lib/galera/galera/src/trx_handle.cpp:396
 
#30 0x00007fec49c895b8 in galera::ReplicatorSMM::apply_trx (this=this@entry=0x55bf53dadd70, recv_ctx=recv_ctx@entry=0x7fec43f68a90, ts=...) at /home/jan/work/galera-lib/galera/galera/src/replicator_smm.cpp:518
 
#31 0x00007fec49c8db63 in galera::ReplicatorSMM::process_trx (this=0x55bf53dadd70, recv_ctx=0x7fec43f68a90, ts_ptr=...) at /home/jan/work/galera-lib/galera/galera/src/replicator_smm.cpp:2152
 
#32 0x00007fec49cc4131 in galera::GcsActionSource::process_writeset (this=0x55bf53df0620, recv_ctx=0x7fec43f68a90, act=..., exit_loop=@0x7fec43f6867f: false) at /home/jan/work/galera-lib/galera/galera/src/gcs_action_source.cpp:62
 
#33 0x00007fec49cc5286 in galera::GcsActionSource::process (this=0x55bf53df0620, recv_ctx=0x7fec43f68a90, exit_loop=@0x7fec43f6867f: false) at /home/jan/work/galera-lib/galera/galera/src/gcs_action_source.cpp:186
 
#34 0x00007fec49c8e2b0 in galera::ReplicatorSMM::async_recv (this=0x55bf53dadd70, recv_ctx=0x7fec43f68a90) at /home/jan/work/galera-lib/galera/galera/src/replicator_smm.cpp:404
 
#35 0x00007fec49c62f8f in galera_recv (gh=<optimized out>, recv_ctx=<optimized out>) at /home/jan/work/galera-lib/galera/galera/src/wsrep_provider.cpp:264
 
#36 0x000055bf52383a88 in wsrep::wsrep_provider_v26::run_applier (this=0x55bf53d98730, applier_ctx=0x7fec43f68a90) at /home/jan/work/mariadb/10.11-backup/wsrep-lib/src/wsrep_provider_v26.cpp:858
 
#37 0x000055bf51d3b7a0 in wsrep_replication_process (thd=0x7feb08000dc8, arg=0x55bf54045cf0) at /home/jan/work/mariadb/10.11-backup/sql/wsrep_thd.cc:57
 
#38 0x000055bf51d28a3f in start_wsrep_THD (arg=0x55bf54045cf0) at /home/jan/work/mariadb/10.11-backup/sql/wsrep_mysqld.cc:3776
 
#39 0x000055bf51c8ee50 in pfs_spawn_thread (arg=0x55bf5453f278) at /home/jan/work/mariadb/10.11-backup/storage/perfschema/pfs.cc:2201
 
#40 0x00007fec4a097ada in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:444
 
#41 0x00007fec4a12847c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
 
 
 
Thread 7 (Thread 0x7fec43fb46c0 (LWP 1334050)):
 
#0  0x00007fec4a0435ca in __GI___sigtimedwait (set=0x7fec43fb3af0, info=0x7fec43fb3a10, timeout=0x0) at ../sysdeps/unix/sysv/linux/sigtimedwait.c:31
 
#1  0x000055bf5139edba in my_sigwait (set=0x7fec43fb3af0, sig=0x7fec43fb3ab4, code=0x7fec43fb3ab8) at /home/jan/work/mariadb/10.11-backup/include/my_pthread.h:192
 
#2  0x000055bf513a5a67 in signal_hand (arg=0x0) at /home/jan/work/mariadb/10.11-backup/sql/mysqld.cc:3274
 
#3  0x000055bf51c8ee50 in pfs_spawn_thread (arg=0x55bf54419f48) at /home/jan/work/mariadb/10.11-backup/storage/perfschema/pfs.cc:2201
 
#4  0x00007fec4a097ada in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:444
 
#5  0x00007fec4a12847c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
 
 
 
Thread 6 (Thread 0x7fec43ed36c0 (LWP 1334055)):
 
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
 
#1  0x000055bf51fe9d27 in srw_mutex_impl<false>::wait (this=0x55bf54057968, lk=2147483653) at /home/jan/work/mariadb/10.11-backup/storage/innobase/sync/srw_lock.cc:244
 
#2  0x000055bf51fe8e43 in srw_mutex_impl<false>::wait_and_lock (this=0x55bf54057968) at /home/jan/work/mariadb/10.11-backup/storage/innobase/sync/srw_lock.cc:331
 
#3  0x000055bf51da4acb in srw_mutex_impl<false>::wr_lock (this=0x55bf54057968) at /home/jan/work/mariadb/10.11-backup/storage/innobase/include/srw_lock.h:132
 
#4  0x000055bf51fe981a in ssux_lock_impl<false>::rd_wait (this=0x55bf54057968) at /home/jan/work/mariadb/10.11-backup/storage/innobase/sync/srw_lock.cc:397
 
#5  0x000055bf51da459d in ssux_lock_impl<false>::rd_lock (this=0x55bf54057968) at /home/jan/work/mariadb/10.11-backup/storage/innobase/include/srw_lock.h:245
 
#6  0x000055bf51f4a894 in ssux_lock::rd_lock (this=0x55bf54057960, file=0x55bf527bbf08 "/home/jan/work/mariadb/10.11-backup/storage/innobase/btr/btr0cur.cc", line=1156) at /home/jan/work/mariadb/10.11-backup/storage/innobase/include/srw_lock.h:433
 
#7  0x000055bf51f4a9c7 in sux_lock<ssux_lock>::s_lock (this=0x55bf54057960, file=0x55bf527bbf08 "/home/jan/work/mariadb/10.11-backup/storage/innobase/btr/btr0cur.cc", line=1156) at /home/jan/work/mariadb/10.11-backup/storage/innobase/include/sux_lock.h:309
 
#8  0x000055bf51f4b2ab in mtr_t::s_lock (this=0x7fec43ed0bb0, file=0x55bf527bbf08 "/home/jan/work/mariadb/10.11-backup/storage/innobase/btr/btr0cur.cc", line=1156, lock=0x55bf54057960) at /home/jan/work/mariadb/10.11-backup/storage/innobase/include/mtr0mtr.h:261
 
#9  0x000055bf52070007 in btr_cur_t::search_leaf (this=0x7fec43ed0950, tuple=0x7feb0c028938, mode=PAGE_CUR_LE, latch_mode=BTR_MODIFY_LEAF, mtr=0x7fec43ed0bb0) at /home/jan/work/mariadb/10.11-backup/storage/innobase/btr/btr0cur.cc:1156
 
#10 0x000055bf51d9ff72 in btr_pcur_open (tuple=0x7feb0c028938, mode=PAGE_CUR_LE, latch_mode=BTR_DELETE_MARK_LEAF, cursor=0x7fec43ed0950, mtr=0x7fec43ed0bb0) at /home/jan/work/mariadb/10.11-backup/storage/innobase/include/btr0pcur.h:431
 
#11 0x000055bf51f9db8c in row_search_index_entry (entry=0x7feb0c028938, mode=BTR_DELETE_MARK_LEAF, pcur=0x7fec43ed0950, mtr=0x7fec43ed0bb0) at /home/jan/work/mariadb/10.11-backup/storage/innobase/row/row0row.cc:1281
 
#12 0x000055bf51fc828d in row_upd_sec_index_entry (node=0x7feb0c025be8, thr=0x7feb0c026018) at /home/jan/work/mariadb/10.11-backup/storage/innobase/row/row0upd.cc:1908
 
#13 0x000055bf51fc8e20 in row_upd_sec_step (node=0x7feb0c025be8, thr=0x7feb0c026018) at /home/jan/work/mariadb/10.11-backup/storage/innobase/row/row0upd.cc:2059
 
#14 0x000055bf51fcbd50 in row_upd (node=0x7feb0c025be8, thr=0x7feb0c026018) at /home/jan/work/mariadb/10.11-backup/storage/innobase/row/row0upd.cc:2806
 
#15 0x000055bf51fcc1e3 in row_upd_step (thr=0x7feb0c026018) at /home/jan/work/mariadb/10.11-backup/storage/innobase/row/row0upd.cc:2921
 
#16 0x000055bf51f70337 in row_update_for_mysql (prebuilt=0x7feb0c025448) at /home/jan/work/mariadb/10.11-backup/storage/innobase/row/row0mysql.cc:1682
 
#17 0x000055bf51d7b9f9 in ha_innobase::update_row (this=0x7feb0c024710, old_row=0x7feb0c024348 "\377*\375\021", new_row=0x7feb0c024288 "\377*\375\021") at /home/jan/work/mariadb/10.11-backup/storage/innobase/handler/ha_innodb.cc:8625
 
#18 0x000055bf5196903d in handler::ha_update_row (this=0x7feb0c024710, old_data=0x7feb0c024348 "\377*\375\021", new_data=0x7feb0c024288 "\377*\375\021") at /home/jan/work/mariadb/10.11-backup/sql/handler.cc:7722
 
#19 0x000055bf51b14e56 in Update_rows_log_event::do_exec_row (this=0x7feb0c022c38, rgi=0x7feb0c021950) at /home/jan/work/mariadb/10.11-backup/sql/log_event_server.cc:8922
 
#20 0x000055bf51b0b7a3 in Rows_log_event::do_apply_event (this=0x7feb0c022c38, rgi=0x7feb0c021950) at /home/jan/work/mariadb/10.11-backup/sql/log_event_server.cc:6146
 
#21 0x000055bf51af12ca in Log_event::apply_event (this=0x7feb0c022c38, rgi=0x7feb0c021950) at /home/jan/work/mariadb/10.11-backup/sql/log_event.cc:4218
 
#22 0x000055bf51d3a88e in wsrep_apply_events (thd=0x7feb0c000dc8, rli=0x7feb0c015a40, events_buf=0x7fec3c14f1d8, buf_len=1502) at /home/jan/work/mariadb/10.11-backup/sql/wsrep_applier.cc:213
 
#23 0x000055bf51d126ed in apply_events (thd=0x7feb0c000dc8, rli=0x7feb0c015a40, data=..., err=...) at /home/jan/work/mariadb/10.11-backup/sql/wsrep_high_priority_service.cc:128
 
#24 0x000055bf51d14c08 in Wsrep_applier_service::apply_write_set (this=0x7fec43ed2a90, ws_meta=..., data=..., err=...) at /home/jan/work/mariadb/10.11-backup/sql/wsrep_high_priority_service.cc:596
 
#25 0x000055bf52366690 in apply_write_set (server_state=..., high_priority_service=..., ws_handle=..., ws_meta=..., data=...) at /home/jan/work/mariadb/10.11-backup/wsrep-lib/src/server_state.cpp:332
 
#26 0x000055bf5236aa66 in wsrep::server_state::on_apply (this=0x55bf53d97d60, high_priority_service=..., ws_handle=..., ws_meta=..., data=...) at /home/jan/work/mariadb/10.11-backup/wsrep-lib/src/server_state.cpp:1128
 
#27 0x000055bf523859ad in wsrep::high_priority_service::apply (this=0x7fec43ed2a90, ws_handle=..., ws_meta=..., data=...) at /home/jan/work/mariadb/10.11-backup/wsrep-lib/include/wsrep/high_priority_service.hpp:47
 
#28 0x000055bf523823cd in (anonymous namespace)::apply_cb (ctx=0x7fec43ed2a90, wsh=0x7fec43ed1cc0, flags=65, buf=0x7fec43ed1cd0, meta=0x7fec43ed1f90, exit_loop=0x7fec43ed1f4f) at /home/jan/work/mariadb/10.11-backup/wsrep-lib/src/wsrep_provider_v26.cpp:507
 
#29 0x00007fec49c76932 in galera::TrxHandleSlave::apply (this=this@entry=0x7feb0c022600, recv_ctx=recv_ctx@entry=0x7fec43ed2a90, apply_cb=0x55bf52382191 <(anonymous namespace)::apply_cb(void*, wsrep_ws_handle_t const*, uint32_t, wsrep_buf_t const*, wsrep_trx_meta_t const*, wsrep_bool_t*)>, meta=..., exit_loop=exit_loop@entry=@0x7fec43ed1f4f: false) at /home/jan/work/galera-lib/galera/galera/src/trx_handle.cpp:396
 
#30 0x00007fec49c895b8 in galera::ReplicatorSMM::apply_trx (this=this@entry=0x55bf53dadd70, recv_ctx=recv_ctx@entry=0x7fec43ed2a90, ts=...) at /home/jan/work/galera-lib/galera/galera/src/replicator_smm.cpp:518
 
#31 0x00007fec49c8db63 in galera::ReplicatorSMM::process_trx (this=0x55bf53dadd70, recv_ctx=0x7fec43ed2a90, ts_ptr=...) at /home/jan/work/galera-lib/galera/galera/src/replicator_smm.cpp:2152
 
#32 0x00007fec49cc4131 in galera::GcsActionSource::process_writeset (this=0x55bf53df0620, recv_ctx=0x7fec43ed2a90, act=..., exit_loop=@0x7fec43ed267f: false) at /home/jan/work/galera-lib/galera/galera/src/gcs_action_source.cpp:62
 
#33 0x00007fec49cc5286 in galera::GcsActionSource::process (this=0x55bf53df0620, recv_ctx=0x7fec43ed2a90, exit_loop=@0x7fec43ed267f: false) at /home/jan/work/galera-lib/galera/galera/src/gcs_action_source.cpp:186
 
#34 0x00007fec49c8e2b0 in galera::ReplicatorSMM::async_recv (this=0x55bf53dadd70, recv_ctx=0x7fec43ed2a90) at /home/jan/work/galera-lib/galera/galera/src/replicator_smm.cpp:404
 
#35 0x00007fec49c62f8f in galera_recv (gh=<optimized out>, recv_ctx=<optimized out>) at /home/jan/work/galera-lib/galera/galera/src/wsrep_provider.cpp:264
 
#36 0x000055bf52383a88 in wsrep::wsrep_provider_v26::run_applier (this=0x55bf53d98730, applier_ctx=0x7fec43ed2a90) at /home/jan/work/mariadb/10.11-backup/wsrep-lib/src/wsrep_provider_v26.cpp:858
 
#37 0x000055bf51d3b7a0 in wsrep_replication_process (thd=0x7feb0c000dc8, arg=0x55bf544e7f60) at /home/jan/work/mariadb/10.11-backup/sql/wsrep_thd.cc:57
 
#38 0x000055bf51d28a3f in start_wsrep_THD (arg=0x55bf544e7f60) at /home/jan/work/mariadb/10.11-backup/sql/wsrep_mysqld.cc:3776
 
#39 0x000055bf51c8ee50 in pfs_spawn_thread (arg=0x55bf5454b998) at /home/jan/work/mariadb/10.11-backup/storage/perfschema/pfs.cc:2201
 
#40 0x00007fec4a097ada in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:444
 
#41 0x00007fec4a12847c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
 
 
 
Thread 5 (Thread 0x7feb3e3d56c0 (LWP 1334045)):
 
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
 
#1  0x000055bf51fe9d73 in srw_mutex_impl<true>::wait (this=0x55bf5397e588 <lock_sys+72>, lk=2147483650) at /home/jan/work/mariadb/10.11-backup/storage/innobase/sync/srw_lock.cc:244
 
#2  0x000055bf51fe90d9 in srw_mutex_impl<true>::wait_and_lock (this=0x55bf5397e588 <lock_sys+72>) at /home/jan/work/mariadb/10.11-backup/storage/innobase/sync/srw_lock.cc:331
 
#3  0x000055bf51da5ca9 in srw_mutex_impl<true>::wr_lock (this=0x55bf5397e588 <lock_sys+72>) at /home/jan/work/mariadb/10.11-backup/storage/innobase/include/srw_lock.h:132
 
#4  0x000055bf51fe97d6 in ssux_lock_impl<true>::rd_wait (this=0x55bf5397e588 <lock_sys+72>) at /home/jan/work/mariadb/10.11-backup/storage/innobase/sync/srw_lock.cc:397
 
#5  0x000055bf51e2c6ef in ssux_lock_impl<true>::rd_lock (this=0x55bf5397e588 <lock_sys+72>) at /home/jan/work/mariadb/10.11-backup/storage/innobase/include/srw_lock.h:245
 
#6  0x000055bf51e518dc in srw_lock_impl<true>::rd_lock (this=0x55bf5397e580 <lock_sys+64>, file=0x55bf5271c540 "/home/jan/work/mariadb/10.11-backup/storage/innobase/lock/lock0lock.cc", line=252) at /home/jan/work/mariadb/10.11-backup/storage/innobase/include/srw_lock.h:513
 
#7  0x000055bf51e32b0e in lock_sys_t::rd_lock (this=0x55bf5397e540 <lock_sys>, file=0x55bf5271c540 "/home/jan/work/mariadb/10.11-backup/storage/innobase/lock/lock0lock.cc", line=252) at /home/jan/work/mariadb/10.11-backup/storage/innobase/lock/lock0lock.cc:443
 
#8  0x000055bf51e31f84 in TMLockGuard::TMLockGuard (this=0x7feb3e3d2fb0, hash=..., id=...) at /home/jan/work/mariadb/10.11-backup/storage/innobase/lock/lock0lock.cc:252
 
#9  0x000055bf51e3af9f in lock_move_reorganize_page (block=0x7febfd01ad00, oblock=0x7febfd01b2b0) at /home/jan/work/mariadb/10.11-backup/storage/innobase/lock/lock0lock.cc:2554
 
#10 0x000055bf520479b6 in btr_page_reorganize_low (cursor=0x7feb3e3d30c0, mtr=0x7feb3e3d3b90) at /home/jan/work/mariadb/10.11-backup/storage/innobase/btr/btr0btr.cc:1510
 
#11 0x000055bf52048760 in btr_page_reorganize_block (z_level=6, block=0x7febfd01ad00, index=0x55bf540577e8, mtr=0x7feb3e3d3b90) at /home/jan/work/mariadb/10.11-backup/storage/innobase/btr/btr0btr.cc:1664
 
#12 0x000055bf52056dc2 in btr_can_merge_with_page (cursor=0x7feb3e3d3930, page_no=278899, merge_block=0x7feb3e3d31d0, mtr=0x7feb3e3d3b90) at /home/jan/work/mariadb/10.11-backup/storage/innobase/btr/btr0btr.cc:5444
 
#13 0x000055bf52050e13 in btr_compress (cursor=0x7feb3e3d3930, adjust=false, mtr=0x7feb3e3d3b90) at /home/jan/work/mariadb/10.11-backup/storage/innobase/btr/btr0btr.cc:3802
 
#14 0x000055bf52080740 in btr_cur_compress_if_useful (cursor=0x7feb3e3d3930, adjust=false, mtr=0x7feb3e3d3b90) at /home/jan/work/mariadb/10.11-backup/storage/innobase/btr/btr0cur.cc:4484
 
#15 0x000055bf520828f7 in btr_cur_pessimistic_delete (err=0x7feb3e3d391c, has_reserved_extents=0, cursor=0x7feb3e3d3930, flags=0, rollback=false, mtr=0x7feb3e3d3b90) at /home/jan/work/mariadb/10.11-backup/storage/innobase/btr/btr0cur.cc:4944
 
#16 0x000055bf521cabac in row_undo_mod_del_mark_or_remove_sec_low (node=0x7feb30001a58, thr=0x7feb30001868, index=0x55bf540577e8, entry=0x7feb30006cd8, mode=BTR_PURGE_TREE) at /home/jan/work/mariadb/10.11-backup/storage/innobase/row/row0umod.cc:593
 
#17 0x000055bf521cacf7 in row_undo_mod_del_mark_or_remove_sec (node=0x7feb30001a58, thr=0x7feb30001868, index=0x55bf540577e8, entry=0x7feb30006cd8) at /home/jan/work/mariadb/10.11-backup/storage/innobase/row/row0umod.cc:638
 
#18 0x000055bf521cbf20 in row_undo_mod_upd_exist_sec (node=0x7feb30001a58, thr=0x7feb30001868) at /home/jan/work/mariadb/10.11-backup/storage/innobase/row/row0umod.cc:1031
 
#19 0x000055bf521ccb62 in row_undo_mod (node=0x7feb30001a58, thr=0x7feb30001868) at /home/jan/work/mariadb/10.11-backup/storage/innobase/row/row0umod.cc:1233
 
#20 0x000055bf51fbd936 in row_undo (node=0x7feb30001a58, thr=0x7feb30001868) at /home/jan/work/mariadb/10.11-backup/storage/innobase/row/row0undo.cc:399
 
#21 0x000055bf51fbdb3a in row_undo_step (thr=0x7feb30001868) at /home/jan/work/mariadb/10.11-backup/storage/innobase/row/row0undo.cc:440
 
#22 0x000055bf51efe41f in que_thr_step (thr=0x7feb30001868) at /home/jan/work/mariadb/10.11-backup/storage/innobase/que/que0que.cc:586
 
#23 0x000055bf51efe6dc in que_run_threads_low (thr=0x7feb30001868) at /home/jan/work/mariadb/10.11-backup/storage/innobase/que/que0que.cc:644
 
#24 0x000055bf51efe812 in que_run_threads (thr=0x7feb30001868) at /home/jan/work/mariadb/10.11-backup/storage/innobase/que/que0que.cc:664
 
#25 0x000055bf52014f20 in trx_rollback_active (trx=0x7fec23405980) at /home/jan/work/mariadb/10.11-backup/storage/innobase/trx/trx0roll.cc:615
 
#26 0x000055bf52015836 in trx_rollback_recovered (all=true) at /home/jan/work/mariadb/10.11-backup/storage/innobase/trx/trx0roll.cc:753
 
#27 0x000055bf52015b86 in trx_rollback_all_recovered () at /home/jan/work/mariadb/10.11-backup/storage/innobase/trx/trx0roll.cc:799
 
#28 0x000055bf521dfaee in tpool::task_group::execute (this=0x55bf539a34c0 <rollback_all_recovered_group>, t=0x55bf539a3560 <rollback_all_recovered_task>) at /home/jan/work/mariadb/10.11-backup/tpool/task_group.cc:70
 
#29 0x000055bf521dfe2a in tpool::task::execute (this=0x55bf539a3560 <rollback_all_recovered_task>) at /home/jan/work/mariadb/10.11-backup/tpool/task.cc:32
 
#30 0x000055bf521d9403 in tpool::thread_pool_generic::worker_main (this=0x55bf53e4a3f0, thread_var=0x55bf53ee5360) at /home/jan/work/mariadb/10.11-backup/tpool/tpool_generic.cc:583
 
#31 0x000055bf521df8d0 in std::__invoke_impl<void, void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> (__f=@0x55bf5406dad8: (void (tpool::thread_pool_generic::*)(tpool::thread_pool_generic * const, tpool::worker_data *)) 0x55bf521d9312 <tpool::thread_pool_generic::worker_main(tpool::worker_data*)>, __t=@0x55bf5406dad0: 0x55bf53e4a3f0) at /usr/include/c++/13/bits/invoke.h:74
 
#32 0x000055bf521df7d6 in std::__invoke<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> (__fn=@0x55bf5406dad8: (void (tpool::thread_pool_generic::*)(tpool::thread_pool_generic * const, tpool::worker_data *)) 0x55bf521d9312 <tpool::thread_pool_generic::worker_main(tpool::worker_data*)>) at /usr/include/c++/13/bits/invoke.h:96
 
#33 0x000055bf521df6f5 in std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> >::_M_invoke<0ul, 1ul, 2ul> (this=0x55bf5406dac8) at /usr/include/c++/13/bits/std_thread.h:292
 
#34 0x000055bf521df68e in std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> >::operator() (this=0x55bf5406dac8) at /usr/include/c++/13/bits/std_thread.h:299
 
#35 0x000055bf521df66e in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> > >::_M_run (this=0x55bf5406dac0) at /usr/include/c++/13/bits/std_thread.h:244
 
#36 0x00007fec4a4e6333 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
 
#37 0x00007fec4a097ada in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:444
 
#38 0x00007fec4a12847c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
 
 
 
Thread 4 (Thread 0x7fec23fff6c0 (LWP 1334042)):
 
#0  0x00007fec4a094076 in __futex_abstimed_wait_common64 (private=1378352235, cancel=true, abstime=0x7fec23ffeb10, op=393, expected=0, futex_word=0x55bf53928a68 <COND_checkpoint+40>) at ./nptl/futex-internal.c:57
 
#1  __futex_abstimed_wait_common (cancel=true, private=1378352235, abstime=0x7fec23ffeb10, clockid=603973696, expected=0, futex_word=0x55bf53928a68 <COND_checkpoint+40>) at ./nptl/futex-internal.c:87
 
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x55bf53928a68 <COND_checkpoint+40>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x7fec23ffeb10, private=private@entry=0) at ./nptl/futex-internal.c:139
 
#3  0x00007fec4a096e7a in __pthread_cond_wait_common (abstime=0x7fec23ffeb10, clockid=0, mutex=0x55bf539289a8 <LOCK_checkpoint+40>, cond=0x55bf53928a40 <COND_checkpoint>) at ./nptl/pthread_cond_wait.c:503
 
#4  ___pthread_cond_timedwait64 (cond=0x55bf53928a40 <COND_checkpoint>, mutex=0x55bf539289a8 <LOCK_checkpoint+40>, abstime=0x7fec23ffeb10) at ./nptl/pthread_cond_wait.c:652
 
#5  0x000055bf522874eb in safe_cond_timedwait (cond=0x55bf53928a40 <COND_checkpoint>, mp=0x55bf53928980 <LOCK_checkpoint>, abstime=0x7fec23ffeb10, file=0x55bf5264e630 "/home/jan/work/mariadb/10.11-backup/include/mysql/psi/mysql_thread.h", line=1088) at /home/jan/work/mariadb/10.11-backup/mysys/thr_mutex.c:548
 
#6  0x000055bf51bdba89 in inline_mysql_cond_timedwait (that=0x55bf53928a40 <COND_checkpoint>, mutex=0x55bf53928980 <LOCK_checkpoint>, abstime=0x7fec23ffeb10, src_file=0x55bf5264e678 "/home/jan/work/mariadb/10.11-backup/storage/maria/ma_servicethread.c", src_line=115) at /home/jan/work/mariadb/10.11-backup/include/mysql/psi/mysql_thread.h:1088
 
#7  0x000055bf51bdbfda in my_service_thread_sleep (control=0x55bf52edd8a0 <checkpoint_control>, sleep_time=29000000000) at /home/jan/work/mariadb/10.11-backup/storage/maria/ma_servicethread.c:115
 
#8  0x000055bf51bcf630 in ma_checkpoint_background (arg=0x1e) at /home/jan/work/mariadb/10.11-backup/storage/maria/ma_checkpoint.c:725
 
#9  0x00007fec4a097ada in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:444
 
#10 0x00007fec4a12847c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
 
 
 
Thread 3 (Thread 0x7fec4b19f6c0 (LWP 1332404)):
 
#0  0x00007fec4a094076 in __futex_abstimed_wait_common64 (private=1378554114, cancel=true, abstime=0x7fec4b19ebc0, op=393, expected=0, futex_word=0x55bf539bc3cc <COND_timer+44>) at ./nptl/futex-internal.c:57
 
#1  __futex_abstimed_wait_common (cancel=true, private=1378554114, abstime=0x7fec4b19ebc0, clockid=1140854000, expected=0, futex_word=0x55bf539bc3cc <COND_timer+44>) at ./nptl/futex-internal.c:87
 
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x55bf539bc3cc <COND_timer+44>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x7fec4b19ebc0, private=private@entry=0) at ./nptl/futex-internal.c:139
 
#3  0x00007fec4a096e7a in __pthread_cond_wait_common (abstime=0x7fec4b19ebc0, clockid=0, mutex=0x55bf539bc308 <LOCK_timer+40>, cond=0x55bf539bc3a0 <COND_timer>) at ./nptl/pthread_cond_wait.c:503
 
#4  ___pthread_cond_timedwait64 (cond=0x55bf539bc3a0 <COND_timer>, mutex=0x55bf539bc308 <LOCK_timer+40>, abstime=0x7fec4b19ebc0) at ./nptl/pthread_cond_wait.c:652
 
#5  0x000055bf522874eb in safe_cond_timedwait (cond=0x55bf539bc3a0 <COND_timer>, mp=0x55bf539bc2e0 <LOCK_timer>, abstime=0x7fec4b19ebc0, file=0x55bf528440c0 "/home/jan/work/mariadb/10.11-backup/include/mysql/psi/mysql_thread.h", line=1088) at /home/jan/work/mariadb/10.11-backup/mysys/thr_mutex.c:548
 
#6  0x000055bf5228858c in inline_mysql_cond_timedwait (that=0x55bf539bc3a0 <COND_timer>, mutex=0x55bf539bc2e0 <LOCK_timer>, abstime=0x7fec4b19ebc0, src_file=0x55bf52844108 "/home/jan/work/mariadb/10.11-backup/mysys/thr_timer.c", src_line=321) at /home/jan/work/mariadb/10.11-backup/include/mysql/psi/mysql_thread.h:1088
 
#7  0x000055bf52289316 in timer_handler (arg=0x0) at /home/jan/work/mariadb/10.11-backup/mysys/thr_timer.c:321
 
#8  0x00007fec4a097ada in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:444
 
#9  0x00007fec4a12847c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
 
 
 
Thread 2 (Thread 0x7fec4ad12540 (LWP 1332403)):
 
#0  0x00007fec4a11a18f in __GI___poll (fds=0x55bf54416d88, nfds=3, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29
 
#1  0x000055bf513ac618 in handle_connections_sockets () at /home/jan/work/mariadb/10.11-backup/sql/mysqld.cc:6307
 
#2  0x000055bf513abb73 in mysqld_main (argc=28, argv=0x55bf53ceb530) at /home/jan/work/mariadb/10.11-backup/sql/mysqld.cc:6012
 
#3  0x000055bf5139ebad in main (argc=2, argv=0x7ffe07667eb8) at /home/jan/work/mariadb/10.11-backup/sql/main.cc:34
 
 
 
Thread 1 (Thread 0x7fec43f1e6c0 (LWP 1334054)):
 
#0  __pthread_kill_implementation (no_tid=0, signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:44
 
#1  __pthread_kill_internal (signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:78
 
#2  __GI___pthread_kill (threadid=<optimized out>, signo=6) at ./nptl/pthread_kill.c:89
 
#3  0x000055bf52281d77 in my_write_core (sig=6) at /home/jan/work/mariadb/10.11-backup/mysys/stacktrace.c:424
 
#4  0x000055bf5194da2d in handle_fatal_signal (sig=6) at /home/jan/work/mariadb/10.11-backup/sql/signal_handler.cc:357
 
#5  <signal handler called>
 
#6  __pthread_kill_implementation (no_tid=0, signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:44
 
#7  __pthread_kill_internal (signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:78
 
#8  __GI___pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
 
#9  0x00007fec4a042866 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
 
#10 0x00007fec4a0268b7 in __GI_abort () at ./stdlib/abort.c:79
 
#11 0x000055bf520383d7 in ut_dbg_assertion_failed (expr=0x55bf5271d172 "v->mysql_thd", file=0x55bf5271c540 "/home/jan/work/mariadb/10.11-backup/storage/innobase/lock/lock0lock.cc", line=1040) at /home/jan/work/mariadb/10.11-backup/storage/innobase/ut/ut0dbg.cc:60
 
#12 0x000055bf51e34cb4 in lock_wait_wsrep (trx=0x7fec23402d80) at /home/jan/work/mariadb/10.11-backup/storage/innobase/lock/lock0lock.cc:1040
 
#13 0x000055bf51e38cb8 in lock_wait (thr=0x7feb00025b48) at /home/jan/work/mariadb/10.11-backup/storage/innobase/lock/lock0lock.cc:1930
 
#14 0x000055bf51f6d4be in row_mysql_handle_errors (new_err=0x7fec43f1b6e8, trx=0x7fec23402d80, thr=0x7feb00025b48, savept=0x0) at /home/jan/work/mariadb/10.11-backup/storage/innobase/row/row0mysql.cc:688
 
#15 0x000055bf51fb5802 in row_search_mvcc (buf=0x7feb00024288 "\377", <incomplete sequence \354>, mode=PAGE_CUR_GE, prebuilt=0x7feb00025448, match_mode=1, direction=0) at /home/jan/work/mariadb/10.11-backup/storage/innobase/row/row0sel.cc:5846
 
#16 0x000055bf51d7ca69 in ha_innobase::index_read (this=0x7feb00024710, buf=0x7feb00024288 "\377", <incomplete sequence \354>, key_ptr=0x7feb00025370 <incomplete sequence \354>, key_len=4, find_flag=HA_READ_KEY_EXACT) at /home/jan/work/mariadb/10.11-backup/storage/innobase/handler/ha_innodb.cc:9044
 
#17 0x000055bf51d7ddd6 in ha_innobase::rnd_pos (this=0x7feb00024710, buf=0x7feb00024288 "\377", <incomplete sequence \354>, pos=0x7feb00025370 <incomplete sequence \354>) at /home/jan/work/mariadb/10.11-backup/storage/innobase/handler/ha_innodb.cc:9527
 
#18 0x000055bf5195a072 in handler::ha_rnd_pos (this=0x7feb00024710, buf=0x7feb00024288 "\377", <incomplete sequence \354>, pos=0x7feb00025370 <incomplete sequence \354>) at /home/jan/work/mariadb/10.11-backup/sql/handler.cc:3495
 
#19 0x000055bf5187a46f in handler::rnd_pos_by_record (this=0x7feb00024710, record=0x7feb00024288 "\377", <incomplete sequence \354>) at /home/jan/work/mariadb/10.11-backup/sql/handler.h:3988
 
#20 0x000055bf517fe0a6 in handler::ha_rnd_pos_by_record (this=0x7feb00024710, buf=0x7feb00024288 "\377", <incomplete sequence \354>) at /home/jan/work/mariadb/10.11-backup/sql/sql_class.h:7584
 
#21 0x000055bf51b12edb in Rows_log_event::find_row (this=0x7feb00022c38, rgi=0x7feb00021950) at /home/jan/work/mariadb/10.11-backup/sql/log_event_server.cc:8356
 
#22 0x000055bf51b147ef in Update_rows_log_event::do_exec_row (this=0x7feb00022c38, rgi=0x7feb00021950) at /home/jan/work/mariadb/10.11-backup/sql/log_event_server.cc:8828
 
#23 0x000055bf51b0b7a3 in Rows_log_event::do_apply_event (this=0x7feb00022c38, rgi=0x7feb00021950) at /home/jan/work/mariadb/10.11-backup/sql/log_event_server.cc:6146
 
#24 0x000055bf51af12ca in Log_event::apply_event (this=0x7feb00022c38, rgi=0x7feb00021950) at /home/jan/work/mariadb/10.11-backup/sql/log_event.cc:4218
 
#25 0x000055bf51d3a88e in wsrep_apply_events (thd=0x7feb00000dc8, rli=0x7feb00015a40, events_buf=0x7fec3c14fa78, buf_len=1505) at /home/jan/work/mariadb/10.11-backup/sql/wsrep_applier.cc:213
 
#26 0x000055bf51d126ed in apply_events (thd=0x7feb00000dc8, rli=0x7feb00015a40, data=..., err=...) at /home/jan/work/mariadb/10.11-backup/sql/wsrep_high_priority_service.cc:128
 
#27 0x000055bf51d14c08 in Wsrep_applier_service::apply_write_set (this=0x7fec43f1da90, ws_meta=..., data=..., err=...) at /home/jan/work/mariadb/10.11-backup/sql/wsrep_high_priority_service.cc:596
 
#28 0x000055bf52366690 in apply_write_set (server_state=..., high_priority_service=..., ws_handle=..., ws_meta=..., data=...) at /home/jan/work/mariadb/10.11-backup/wsrep-lib/src/server_state.cpp:332
 
#29 0x000055bf5236aa66 in wsrep::server_state::on_apply (this=0x55bf53d97d60, high_priority_service=..., ws_handle=..., ws_meta=..., data=...) at /home/jan/work/mariadb/10.11-backup/wsrep-lib/src/server_state.cpp:1128
 
#30 0x000055bf523859ad in wsrep::high_priority_service::apply (this=0x7fec43f1da90, ws_handle=..., ws_meta=..., data=...) at /home/jan/work/mariadb/10.11-backup/wsrep-lib/include/wsrep/high_priority_service.hpp:47
 
#31 0x000055bf523823cd in (anonymous namespace)::apply_cb (ctx=0x7fec43f1da90, wsh=0x7fec43f1ccc0, flags=65, buf=0x7fec43f1ccd0, meta=0x7fec43f1cf90, exit_loop=0x7fec43f1cf4f) at /home/jan/work/mariadb/10.11-backup/wsrep-lib/src/wsrep_provider_v26.cpp:507
 
#32 0x00007fec49c76932 in galera::TrxHandleSlave::apply (this=this@entry=0x7feb00022600, recv_ctx=recv_ctx@entry=0x7fec43f1da90, apply_cb=0x55bf52382191 <(anonymous namespace)::apply_cb(void*, wsrep_ws_handle_t const*, uint32_t, wsrep_buf_t const*, wsrep_trx_meta_t const*, wsrep_bool_t*)>, meta=..., exit_loop=exit_loop@entry=@0x7fec43f1cf4f: false) at /home/jan/work/galera-lib/galera/galera/src/trx_handle.cpp:396
 
#33 0x00007fec49c895b8 in galera::ReplicatorSMM::apply_trx (this=this@entry=0x55bf53dadd70, recv_ctx=recv_ctx@entry=0x7fec43f1da90, ts=...) at /home/jan/work/galera-lib/galera/galera/src/replicator_smm.cpp:518
 
#34 0x00007fec49c8db63 in galera::ReplicatorSMM::process_trx (this=0x55bf53dadd70, recv_ctx=0x7fec43f1da90, ts_ptr=...) at /home/jan/work/galera-lib/galera/galera/src/replicator_smm.cpp:2152
 
#35 0x00007fec49cc4131 in galera::GcsActionSource::process_writeset (this=0x55bf53df0620, recv_ctx=0x7fec43f1da90, act=..., exit_loop=@0x7fec43f1d67f: false) at /home/jan/work/galera-lib/galera/galera/src/gcs_action_source.cpp:62
 
#36 0x00007fec49cc5286 in galera::GcsActionSource::process (this=0x55bf53df0620, recv_ctx=0x7fec43f1da90, exit_loop=@0x7fec43f1d67f: false) at /home/jan/work/galera-lib/galera/galera/src/gcs_action_source.cpp:186
 
#37 0x00007fec49c8e2b0 in galera::ReplicatorSMM::async_recv (this=0x55bf53dadd70, recv_ctx=0x7fec43f1da90) at /home/jan/work/galera-lib/galera/galera/src/replicator_smm.cpp:404
 
#38 0x00007fec49c62f8f in galera_recv (gh=<optimized out>, recv_ctx=<optimized out>) at /home/jan/work/galera-lib/galera/galera/src/wsrep_provider.cpp:264
 
#39 0x000055bf52383a88 in wsrep::wsrep_provider_v26::run_applier (this=0x55bf53d98730, applier_ctx=0x7fec43f1da90) at /home/jan/work/mariadb/10.11-backup/wsrep-lib/src/wsrep_provider_v26.cpp:858
 
#40 0x000055bf51d3b7a0 in wsrep_replication_process (thd=0x7feb00000dc8, arg=0x55bf544d5520) at /home/jan/work/mariadb/10.11-backup/sql/wsrep_thd.cc:57
 
#41 0x000055bf51d28a3f in start_wsrep_THD (arg=0x55bf544d5520) at /home/jan/work/mariadb/10.11-backup/sql/wsrep_mysqld.cc:3776
 
#42 0x000055bf51c8ee50 in pfs_spawn_thread (arg=0x55bf545433b8) at /home/jan/work/mariadb/10.11-backup/storage/perfschema/pfs.cc:2201
 
#43 0x00007fec4a097ada in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:444
 
#44 0x00007fec4a12847c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
 
(gdb)

For some reason I could not find the thread where trx->id would be same as v->id i.e. 22537.

Comment by Marko Mäkelä [ 2024-01-24 ]

I see v->is_recovered=true and v->state.m=TRX_STATE_ACTIVE. I also see that trx_rollback_all_recovered() has not finished executing in Thread 5 yet. If it had, it should have rolled back the transaction v and released all its locks.

I would suggest that you wait for the rollback of recovered transactions to complete before starting wsrep_replication_process().

Comment by Jan Lindström [ 2024-01-24 ]

Unfortunately your suggestion is not feasible for GA-releases because I would need to add new function to SE API to ask value of trx_rollback_is_active from InnoDB. Furthermore, with physical methods (rsync, mariabackup) Galera connects to the cluster and starts one applier thread before SE and dynamic plugins are initialized. Changing that is bigger task.

In my 2-node test case, node_1 contains 20-million rows in sysbench sbtest1 table and sysbench run phase is running against it. Then, node_2 is started from empty datadir i.e. it will do SST. My first stacktrace did contain background statistic calculation but I should have added full stacktrace because you are correct most likely offending transaction was recovered trx. wsrep_init_startup() starts at least 1 applier.

Comment by Marko Mäkelä [ 2024-01-24 ]

Did you consider modifying the Galera-specific code in InnoDB?

diff --git a/storage/innobase/lock/lock0lock.cc b/storage/innobase/lock/lock0lock.cc
 
index df51ceb16d8..19745ebc239 100644
 
--- a/storage/innobase/lock/lock0lock.cc
 
+++ b/storage/innobase/lock/lock0lock.cc
 
@@ -51,6 +51,7 @@ Created 5/7/1996 Heikki Tuuri
 
 #include <set>
 
 
 #ifdef WITH_WSREP
 
+#include "trx0roll.h"
 
 #include <mysql/service_wsrep.h>
 
 #endif /* WITH_WSREP */
 
 
@@ -956,6 +957,9 @@ static void lock_wait_wsrep(trx_t *trx)
 
   if (!wsrep_thd_is_BF(trx->mysql_thd, false))
 
     return;
 
 
+  while (UNIV_UNLIKELY(trx_rollback_is_active))
 
+    std::this_thread::sleep_for(std::chrono::seconds(1));
 
+
 
   std::set<trx_t*> victims;
 
 
   lock_sys.wr_lock(SRW_LOCK_CALL);

The above is somewhat ugly, because there is no mutex or condition variable related to that global variable. The only other place where this variable is being read is logs_empty_and_mark_files_at_shutdown() during shutdown. You’d have to refine this for the case that the server is being shut down while innodb_fast_shutdown=3 (MDEV-15832) or innodb_fast_shutdown=2 has been set.

Comment by Jan Lindström [ 2024-01-25 ]

Yes, I did consider and based on your description suggested change is a lot more complicated and error prone compared to what I did propose. Naturally, most of the time my added condition is not needed but it is not much effecting normal performance.

Comment by Marko Mäkelä [ 2024-01-25 ]

I realized that due to the way how trx_t::commit_in_memory() is updating some data members before invoking trx_t::release_locks(), it is not possible to write debug assertions that would document this scenario in lock_wait_wsrep(). A recovered transaction could be in the process of being rolled back, concurrently with lock_wait_wsrep(). Once the rollback is completed, the locks would be released and the thread that invoked lock_wait_wsrep() to be woken up in lock_wait().

Your initially suggested code change should be correct, but the source code comments as well as the commit message will need to be clarified.

Comment by Jan Lindström [ 2024-01-26 ]

I addressed your comments at https://github.com/MariaDB/server/commit/f7d2bb92d43b0c9298ab6536da99433a543050e9

Comment by Jan Lindström [ 2024-01-29 ]

https://github.com/MariaDB/server/commit/a68e440cf0bf24c3b4afa78911a7b7ffe5690155

Generated at Thu Feb 08 10:37:43 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.