Daniel Black added a comment - 2020-10-03 03:55 ref: https://buildbot.mariadb.org/#/builders/33/builds/4843/steps/8/logs/stdio

Marko Mäkelä added a comment - 2022-10-24 13:48 The hang looks like a latching order violation similar to the one reported in MDEV-29835 . It used to mostly occur on the 32-bit debug builder kvm-fulltest2 . Locally, I can reliably repeat it with an ASAN debug build of 10.6. It seems that code paths that involve rtr_pcur_open() are not properly holding an exclusive index latch. I diagnosed a deadlock between two threads: one performing an INSERT and another executing rtr_get_father_node() inside btr_compress() as part of a purge of transaction history. The latter thread should either hold an exclusive index latch, or it should never have released the parent page latch in the first place.

Marko Mäkelä added a comment - 2022-10-24 14:01 I debugged this as follows: ./mtr --rr --parallel=auto innodb_gis.rtree_purge rr replay var/1/log/mysqld.1.rr/latest-trace I set a breakpoint to the rtr_page_get_father_block() call in btr_compress() and a watchpoint on father_cursor.page_cur.block.page.lock.writer . Then, reverse-continue and backtrace would show where the latch was prematurely released: #1 0x000055ae4f00926e in mtr_t::release_block_at_savepoint (this=<optimized out>, savepoint=<optimized out>, block=<optimized out>) at /mariadb/10.6/storage/innobase/include/mtr0mtr.inl:180 #2 0x000055ae4f0b0df9 in rtr_pcur_getnext_from_path (tuple=tuple@entry=0x7f6bac0050b0, mode=mode@entry=PAGE_CUR_RTREE_LOCATE, btr_cur=btr_cur@entry=0x7f6bb77ecd40, target_level=target_level@entry=0, latch_mode=latch_mode@entry=522, index_locked=<optimized out>, mtr=<optimized out>) at /mariadb/10.6/storage/innobase/gis/gis0sea.cc:232 #3 0x000055ae4f0a96e8 in rtr_pcur_open (index=index@entry=0x7f6b900ab2a0, tuple=<optimized out>, tuple@entry=0x7f6bac0050b0, latch_mode=latch_mode@entry=522, cursor=cursor@entry=0x7f6bb77ecd40, mtr=mtr@entry=0x7f6bb77ecfe0) at /mariadb/10.6/storage/innobase/gis/gis0sea.cc:593 #4 0x000055ae4f25233e in row_search_index_entry (index=index@entry=0x7f6b900ab2a0, entry=entry@entry=0x7f6bac0050b0, mode=mode@entry=522, pcur=pcur@entry=0x7f6bb77ecd40, mtr=mtr@entry=0x7f6bb77ecfe0) at /mariadb/10.6/storage/innobase/row/row0row.cc:1300 There is a confusing comment in rtr_pcur_getnext_from_path() : /* Once we have pages in "path", these pages are predicate page locked, so they can't be shrunk away. They also have SSN (split sequence number) to detect splits, so we can directly latch single page while getting them. They can be unlatched if not qualified. One reason for pre-latch is that we might need to position some parent position (requires latch) during search */ After this comment, we release all non-leaf page latches, which would seem to be a prerequisite for this hang. There is a FIXME comment referring to MDEV-14059 , noting that for some reason, the SPATIAL INDEX code is making copies of block descriptors. The dodgy latching and locking rules for SPATIAL INDEX that we adopted from MySQL 5.7 could easily explain intermittent-wrong-result bugs like MDEV-15284 .

Daniel Black added a comment - 2024-07-12 03:04 10.5-ea9869504d4bd47e111fe2aab99f85b60946f648 , url: https://buildbot.mariadb.org/#/builders/677/builds/1150/steps/6/logs/stdio innodb_gis.rtree_purge '64k' w12 [ fail ] timeout after 900 seconds Test ended at 2024-07-11 11:26:52 Test case timeout after 900 seconds == /home/buildbot/s390x-ubuntu-2004-debug/build/mysql-test/var/12/log/rtree_purge.log == SET @saved_frequency = @@GLOBAL.innodb_purge_rseg_truncate_frequency; SET GLOBAL innodb_purge_rseg_truncate_frequency = 1; create table t ( b point not null,d point not null, spatial key (d),spatial key (b) ) engine=innodb; == /home/buildbot/s390x-ubuntu-2004-debug/build/mysql-test/var/12/tmp/analyze-timeout-mysqld.1.err == mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local MySQL server through socket '/home/buildbot/s390x-ubuntu-2004-debug/build/mysql-test/var/tmp/' (111) .. Core generated by '/home/buildbot/s390x-ubuntu-2004-debug/build/sql/mariadbd' Output from gdb follows. The first stack trace is from the failing thread. The following stack traces are from all threads (so the failing one is duplicated). -------------------------- [New LWP 179295] [New LWP 179331] [New LWP 179330] [New LWP 179401] [New LWP 179323] [New LWP 202701] [New LWP 179367] [New LWP 179386] [New LWP 179327] [New LWP 195526] [New LWP 179409] [New LWP 202702] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/s390x-linux-gnu/libthread_db.so.1". Core was generated by `/home/buildbot/s390x-ubuntu-2004-debug/build/sql/mariadbd --defaults-group-suff'. Program terminated with signal SIGABRT, Aborted. #0 0x000003ff8729029e in pthread_kill () from /lib/s390x-linux-gnu/libpthread.so.0 [Current thread is 1 (Thread 0x3ff87d6b4d0 (LWP 179295))] #0 0x000003ff8729029e in pthread_kill () from /lib/s390x-linux-gnu/libpthread.so.0 #1 0x000002aa39d6613a in my_write_core (sig=6) at /home/buildbot/s390x-ubuntu-2004-debug/build/mysys/stacktrace.c:424 #2 0x000002aa39209be6 in handle_fatal_signal (sig=6) at /home/buildbot/s390x-ubuntu-2004-debug/build/sql/signal_handler.cc:357 #3 <signal handler called> #4 0x000003ff86cf1c92 in poll () from /lib/s390x-linux-gnu/libc.so.6 #5 0x000002aa38c302b0 in handle_connections_sockets () at /home/buildbot/s390x-ubuntu-2004-debug/build/sql/mysqld.cc:6305 #6 0x000002aa38c2f7b2 in mysqld_main (argc=167, argv=0x2aa3c4058c0) at /home/buildbot/s390x-ubuntu-2004-debug/build/sql/mysqld.cc:5754 #7 0x000002aa38c218bc in main (argc=26, argv=0x3ffeb17bad8) at /home/buildbot/s390x-ubuntu-2004-debug/build/sql/main.cc:25 Thread 12 (Thread 0x3ff61ffb900 (LWP 202702)): #0 0x000002aa397a5454 in rec_offs_n_fields (offsets=0x3ff61ff7428) at /home/buildbot/s390x-ubuntu-2004-debug/build/storage/innobase/include/rem0rec.h:625 n_fields = 240 __PRETTY_FUNCTION__ = "ulint rec_offs_n_fields(const rec_offs*)" #1 0x000002aa398e8ea6 in rec_get_nth_field_offs (offsets=0x3ff61ff7428, n=0, len=0x3ff61ff7268) at /home/buildbot/s390x-ubuntu-2004-debug/build/storage/innobase/include/rem0rec.inl:723 __PRETTY_FUNCTION__ = "rec_offs rec_get_nth_field_offs(const rec_offs*, ulint, ulint*)" offs = 25087 next_offs = 29736 #2 0x000002aa398eafe8 in cmp_dtuple_rec_with_match_low (dtuple=0x3ff38004758, rec=0x3ff7b294513 "", offsets=0x3ff61ff7428, n_cmp=1, matched_fields=0x3ff61ff73c0) at /home/buildbot/s390x-ubuntu-2004-debug/build/storage/innobase/rem/rem0cmp.cc:407 dtuple_b_ptr = 0x2aa3c82520d "" type = 0x3ff380047a0 rec_f_len = 2930150559616 rec_b_ptr = 0x3ff38004758 "" dtuple_field = 0x3ff38004790 dtuple_f_len = 6 cur_field = 0 ret = 1644130824 __PRETTY_FUNCTION__ = "int cmp_dtuple_rec_with_match_low(const dtuple_t*, const rec_t*, const rec_offs*, ulint, ulint*)" #3 0x000002aa39893fb8 in page_cur_search_with_match (block=0x3ff7a80e970, index=0x3ff20236e88, tuple=0x3ff38004758, mode=PAGE_CUR_LE, iup_matched_fields=0x3ff61ff7c90, ilow_matched_fields=0x3ff61ff7ca0, cursor=0x2aa3c7f9ad0, rtr_info=0x0) at /home/buildbot/s390x-ubuntu-2004-debug/build/storage/innobase/page/page0cur.cc:400 up = 2 low = 0 mid = 1 page = 0x3ff7b290000 "" slot = 0x3ff7b29fff4 "E\023" up_rec = 0x2aa39751bd8 <OSMutex::exit()+176> "\271\004" low_rec = 0x2aa3b4a6e70 <rw_lock_debug_mutex+56> "" mid_rec = 0x3ff7b294513 "" up_matched_fields = 0 low_matched_fields = 0 cur_matched_fields = 0 cmp = -2 heap = 0x0 offsets_ = {300, 1, 0, 1023, 31529, 17683, 0, 1023, 8227, 28296, 32773, 6, 0, 1023, 25087, 47360, 0, 682, 14688, 35836, 0, 1023, 25087, 29656, 0, 682, 14997, 20352, 0, 1023, 25087, 47360, 0, 1023, 25087, 30056, 0, 0, 0, 80, 0, 1023, 34052, 65224, 0, 1023, 0, 0, 0, 1023, 25087, 47360, 0, 1023, 25087, 29752, 0, 682, 14997, 20352, 0, 1023, 25087, 47360, 0, 682, 14709, 7128, 0, 1023, 25087, 29752, 0, 682, 14885, 11860, 0, 1169, 25087, 29800, 0, 1023, 25087, 47360, 256, 1023, 25087, 47360, 0, 1023, 25087, 29824, 0, 682, 14997, 20352, 0, 1023, 25087, 47360, 0, 682, 14689, 36564, 0, 1023, 25087, 29824, 252, 23494, 52632, 57768, 0, 1023, 0, 0, 0, 1023, 25087, 30424, 0, 7, 0, 162, 0, 1023, 34365, 58624, 0, 0, 0, 80, 0, 1023, 0, 178, 0, 1023, 0, 177, 0, 682, 15429, 14336, 0, 1023, 34365, 58624, 0, 1023, 34052, 52480, 0, 1023, 34052, 65224, 0, 0, 0, 0, 0, 682, 14997, 20352, 0, 1023, 25087, 47360, 0, 682, 14709, 7128, 0, 1023, 25087, 29960, 65535, 65535, 65535, 65535, 0, 682, 0 <repeats 11 times>, 682, 15178, 28224, 0, 682, 0, 0, 0...} offsets = 0x3ff61ff7428 __PRETTY_FUNCTION__ = "void page_cur_search_with_match(const buf_block_t*, const dict_index_t*, const dtuple_t*, page_cur_mode_t, ulint*, ulint*, page_cur_t*, rtr_info_t*)" n_core = 5 #4 0x000002aa39ad8256 in btr_cur_search_to_nth_level (index=0x3ff20236e88, level=0, tuple=0x3ff38004758, mode=PAGE_CUR_LE, latch_mode=1, cursor=0x2aa3c7f9ac8, file=0x2aa3a231690 "/home/buildbot/s390x-ubuntu-2004-debug/build/storage/innobase/row/row0row.cc", line=1215, mtr=0x3ff61ff8e78, autoinc=0) at /home/buildbot/s390x-ubuntu-2004-debug/build/storage/innobase/btr/btr0cur.cc:1986 page = 0x3ff7b290000 "" block = 0x3ff7a80e970 guess = 0x0 height = 0 up_match = 0 up_bytes = 0 low_match = 0 low_bytes = 0 rw_latch = 1 page_mode = PAGE_CUR_LE search_mode = PAGE_CUR_UNSUPP buf_mode = 10 estimate = 0 node_ptr_max_size = 32768 page_cursor = 0x2aa3c7f9ad0 btr_op = BTR_NO_OP root_height = 1 err = DB_SUCCESS lock_intention = BTR_INTENTION_BOTH modify_external = false tree_blocks = {0x3ff7a81ec30, 0x3ff7a80e970, 0x3ff61ffb900, 0x2aa396330f4 <sanitize_thread(PFS_thread*)+44>, 0x3ff61ff82a8, 0x3ff61ffb900, 0x3ff61ff82d8, 0x3ff61ff8458, 0x3ff809fc4b7, 0x3ff61ffafe0, 0x2aa3c65e9f0, 0x1c, 0x3ff61ff8308, 0x2aa3a954f80, 0x3ff61ffb900, 0x2aa39ad8296 <btr_cur_search_to_nth_level(dict_index_t*, unsigned long, dtuple_t const*, page_cur_mode_t, unsigned long, btr_cur_t*, char const*, unsigned int, mtr_t*, unsigned long)+13830>, 0x3ff61ff8308, 0x3ff61ff84e0, 0x2aa39a12cae <PolicyMutex<OSTrackMutex<GenericPolicy> >::pfs_end(PSI_mutex_locker*, int)+86>, 0x3ff61ff8320, 0x3ff61ff8618, 0x3ff61ff8468, 0x3ff61ff9430, 0x0, 0x3ff61ff8434, 0x0, 0x2aa3a954f80, 0x3ff61ffb900, 0x3ff61ff84e8, 0x198, 0x3ff8504fad8, 0x2aa39a1482c <MutexDebug<OSTrackMutex<GenericPolicy> >::get() const+172>, 0x2, 0x50000000d, 0x3ff38004da8, 0x0, 0x3ff20069de8, 0x2aa00000000, 0x30000000000, 0x40000000d, 0x1, 0x200000022, 0x261ff83f8, 0x1, 0x0, 0x2, 0x0, 0x0, 0x700000004, 0x3ff7a81ed98, 0x3ff7a81ed98, 0x0, 0x2, 0xa, 0x10, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x3ff61ff8618, 0x3ff61ff8870, 0x3ff61ffb900, 0x10003ff61ffb900, 0x3ff61ff84c0, 0x2aa3a954f80, 0x0, 0x1, 0x3ff7be20000, 0x0, 0x0, 0x3ff200894b8, 0x0, 0x3ff61ff9430, 0x0, 0x2aa3c408790, 0x2aa397634a8 <ut_allocator<unsigned char, true>::allocate_trace(unsigned long, unsigned int, ut_new_pfx_t*)+120>, 0x3ff61ff8508, 0x3ff863de500, 0x3ff8504cd00, 0x3ff8504fec8, 0x0, 0x3ff38005b40, 0x3ff00000022, 0x198, 0x3ff61ff8914, 0xffffffff000000a4, 0x2aa00000000, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x12c03ff863de500} tree_savepoints = {16, 32, 2930178557840, 4396003488000, 4396004929792, 0, 4395395679736, 2930150559616, 4395395692800, 71032603542086056, 34, 0, 4395395679784, 2930150559616, 4395395692800, 2930130497780, 4395395679784, 4395395679824, 2930150559616, 4394691090840, 2930130497780, 4395395679824, 4395395679864, 2930150559616, 2930161657484, 21474836485, 4396003747072, 4395395679904, 2930150559616, 4395395692800, 2930134556350, 0, 4395395680600, 4395395679952, 2930150559616, 4395395692800, 4395395684064, 4395395679984, 4395395679992, 2930150559616, 4394691090840, 2930130324476, 4395395679992, 4395395680032, 2930150559616, 4395395680400, 40, 4395983237376, 2930178557840, 4396003488000, 4396004929792, 0, 2930161843264, 4393751543808, 4395395681016, 2929167695872, 4395395692800, 2930130390740, 4395395680120, 4395395680160, 2930150559616, 4394691090840, 2930130388844, 4395395680160, 2930150559616, 4394691088864, 2930130497780, 4395395680192, 4394691072616, 40, 2929167696001, 21474836608, 4394691072616, 2930178852608, 84446888996424960, 4395983237376, 4396003747072, 0, 4395395690464, 2933462663167, 4394288976864, 4395395680336, 2930150559616, 4395395692800, 4395395680360, 2930150559616, 4394691088864, 2930130324476, 4394691086224, 21474836485, 4396003747072, 4395395680768, 136, 4395395680432, 2930150559616, 4394691090840, 2930137073600, 4395395680432, 2930182691528, 4394290736776} n_blocks = 1 n_releases = 1 detected_same_key_root = false retrying_for_search_prev = false leftmost_from_level = 0 prev_tree_blocks = 0x0 prev_tree_savepoints = 0x0 prev_n_blocks = 0 prev_n_releases = 0 need_path = true rtree_parent_modified = false mbr_adj = false found = false info = 0x3ff20088e70 heap = 0x0 offsets_ = {300, 2, 0, 1023, 31713, 125, 0, 1023, 8227, 28296, 32773, 6, 10, 1023, 25087, 47360, 0, 682, 14715, 43342, 0, 1023, 25087, 32256, 0, 1023, 25087, 32296, 0, 1023, 25087, 32304, 0, 682, 14997, 20352, 0, 1023, 25087, 47360, 252, 23494, 52632, 57768, 0, 0, 128, 4096, 0, 682, 15461, 59888, 0, 0, 0, 1169, 0, 1023, 32927, 50359, 0, 682, 15178, 28216, 0, 682, 15178, 28216, 0, 0, 0, 28, 0, 0, 0, 10, 0, 1023, 32927, 50359, 0, 1023, 25087, 45024, 0, 682, 15461, 59888, 0, 1023, 25087, 47360, 0, 1023, 25087, 32432, 0, 682, 14997, 20352, 0, 1023, 25087, 47360, 0, 682, 14715, 34870, 0, 1023, 25087, 32432, 0, 1023, 25087, 32472, 0, 682, 14997, 20352, 0, 1023, 25087, 47360, 0, 682, 14753, 7870, 0, 0, 0, 0, 0, 1023, 14336, 23144, 0, 1023, 25087, 32796, 0, 1023, 14336, 23120, 0, 65535, 0, 5, 0, 682, 15178, 28216, 0, 682, 15461, 59888, 0, 682, 14794, 26912, 0, 1023, 25087, 32568, 0, 682, 14997, 20352, 0, 1023, 25087, 47360, 0, 682, 14770, 26072, 0, 0, 0, 1169, 0, 1023, 32927, 50359, 0, 1023, 25087, 32616, 0, 1023, 25087, 32624, 0, 682, 14997, 20352, 0, 1023, 25087, 47360...} offsets = 0x3ff61ff7e50 offsets2_ = {300, 1023, 25087, 32816, 0, 1023, 25087, 32824, 0, 1023, 0, 16, 0, 3, 0, 1, 0, 1023, 8198, 40828, 0, 1023, 8198, 40792, 0, 1023, 4096, 0, 0, 1023, 25087, 33360, 0, 1023, 25087, 33352, 0, 1023, 31361, 60832, 0, 65535, 0, 0, 0, 682, 0, 0, 0, 1023, 25087, 33304, 0, 1023, 0, 242, 57184, 29941, 27833, 51938, 0, 0, 0, 8704, 0, 1023, 25087, 32976, 0, 0, 0, 28, 0, 1023, 25087, 32992, 0, 682, 14997, 20352, 0, 1023, 25087, 47360, 0, 682, 14764, 22448, 0, 1023, 25087, 32992, 0, 682, 14709, 62586, 0, 1023, 25087, 33008, 0, 1023, 25087, 33328, 0, 1023, 25087, 38600, 0, 1023, 8198, 40792, 0, 0, 0, 0, 0, 1023, 4095, 65535, 0, 1023, 25087, 33352, 0, 65535, 0, 5, 0, 0, 1, 0, 0, 1023, 31714, 0, 0, 1023, 25087, 47360, 0, 682, 14740, 10136, 0, 1023, 25087, 33136, 0, 682, 14997, 20352, 0, 1023, 25087, 33152, 0, 0, 0, 28, 0, 1023, 25087, 33168, 0, 1023, 25087, 33176, 0, 682, 14997, 20352, 0, 1023, 25087, 47360, 0, 682, 14764, 24266, 0, 1023, 25087, 33208, 0, 682, 14997, 20352, 0, 1023, 25087, 47360, 0, 682, 14714, 24880, 0, 1023, 25087, 33208...} offsets2 = 0x3ff61ff80a8 __PRETTY_FUNCTION__ = "dberr_t btr_cur_search_to_nth_level(dict_index_t*, ulint, const dtuple_t*, page_cur_mode_t, ulint, btr_cur_t*, const char*, unsigned int, mtr_t*, ib_uint64_t)" s_latch_by_caller = 0 savepoint = 0 upper_rw_latch = RW_S_LATCH root_leaf_rw_latch = RW_S_LATCH zip_size = 0 page_id = {m_id = 30064771083} latch_leaves = {blocks = {0x0, 0x0, 0x0}, savepoints = {0, 0, 0}} #5 0x000002aa399aebfa in btr_pcur_open_low (index=0x3ff20236e88, level=0, tuple=0x3ff38004758, mode=PAGE_CUR_LE, latch_mode=1, cursor=0x2aa3c7f9ac8, file=0x2aa3a231690 "/home/buildbot/s390x-ubuntu-2004-debug/build/storage/innobase/row/row0row.cc", line=1215, autoinc=0, mtr=0x3ff61ff8e78) at /home/buildbot/s390x-ubuntu-2004-debug/build/storage/innobase/include/btr0pcur.inl:441 btr_cursor = 0x2aa3c7f9ac8 err = DB_SUCCESS __PRETTY_FUNCTION__ = "dberr_t btr_pcur_open_low(dict_index_t*, ulint, const dtuple_t*, page_cur_mode_t, ulint, btr_pcur_t*, const char*, unsigned int, ib_uint64_t, mtr_t*)" #6 0x000002aa399b371e in row_search_on_row_ref (pcur=0x2aa3c7f9ac8, mode=1, table=0x3ff201657c8, ref=0x3ff38004758, mtr=0x3ff61ff8e78) at /home/buildbot/s390x-ubuntu-2004-debug/build/storage/innobase/row/row0row.cc:1215 low_match = 4393751543808 rec = 0x3ff61ffafe0 "" index = 0x3ff20236e88 __PRETTY_FUNCTION__ = "ulint row_search_on_row_ref(btr_pcur_t*, ulint, const dict_table_t*, const dtuple_t*, mtr_t*)" #7 0x000002aa399a25c6 in row_purge_reposition_pcur (mode=1, node=0x2aa3c7f9a28, mtr=0x3ff61ff8e78) at /home/buildbot/s390x-ubuntu-2004-debug/build/storage/innobase/row/row0purge.cc:81 __PRETTY_FUNCTION__ = "ulint row_purge_reposition_pcur(ulint, purge_node_t*, mtr_t*)" #8 0x000002aa399a2d6e in row_purge_poss_sec (node=0x2aa3c7f9a28, index=0x3ff20069de8, entry=0x3ff38004da8, sec_pcur=0x3ff61ff9428, sec_mtr=0x3ff61ff96b0, is_tree=false) at /home/buildbot/s390x-ubuntu-2004-debug/build/storage/innobase/row/row0purge.cc:248 can_delete = false mtr = {m_start = true, m_commit = false, m_last = 0x0, m_last_offset = 0, m_log_mode = 0, m_modifications = 0, m_made_dirty = 0, m_inside_ibuf = 0, m_trim_pages = 0, m_user_space_id = 0, m_memo = {m_heap = 0x0, m_list = {<ilist<mtr_buf_t::block_t, void>> = {sentinel_ = {next = 0x3ff61ff8eb8, prev = 0x3ff61ff8eb8}}, size_ = 1}, m_size = 48, m_first_block = {<ilist_node<void>> = {next = 0x3ff61ff8e98, prev = 0x3ff61ff8e98}, m_buf_end = 0, m_magic_n = 375767, m_data = '\000' <repeats 11 times>, " 9u\033\330", '\000' <repeats 11 times>, "\001\377\377\377\377\000\000\003\377z\200\351p\000\000\000\001", '\000' <repeats 14 times>, "\002\252;Jn@\000\000\002\252\000\000\000\000\000\000\002\252:\225O\200\000\000\003\377a\377\271\000\000\000\002\252\071\241,\256\000\000\003\377a\377\216\300\001\000\002\252:\225O\200\000\000\003\377a\377\216\360\000\000\002\252:\225O\200\000\000\003\377a\377\271\000\000\000\002\252\071{\251N\000\000\003\377a\377\216\360\000\000\003\377a\377\217\030\000\000\003\377a\377\217 \000\000\002\252:\225O\200\000\000\003\377a\377\271\000\000\374[\306\315\230\341\250\000\000\000\000\000\200\020\000\000\000\000\000\000\000\000"..., m_used = 48}}, m_log = {m_heap = 0x0, m_list = {<ilist<mtr_buf_t::block_t, void>> = {sentinel_ = {next = 0x3ff61ff90f8, prev = 0x3ff61ff90f8}}, size_ = 1}, m_size = 0, m_first_block = {<ilist_node<void>> = {next = 0x3ff61ff90d8, prev = 0x3ff61ff90d8}, m_buf_end = 0, m_magic_n = 375767, m_data = "\000\000\003\377 \006\237\200\000\000\000\000\000\000\000\000\000\000\002\252:\"\325\006\000\000\003\377a\377\220\330\000\000\003\377a\377\220\340\000\000\003\377a\377\226\260\000\000\003\377\200\237\304\267\000\000\003\377a\377\257\340\000\000\002\252<e\351\360\000\000\003\377a\377\271\000\000\000\003\377a\377\221\020\000\000\002\252:\225O\200\000\000\003\377a\377\221 \000\000\002\252:\225O\200\000\000\003\377a\377\271\000\000\000\002\252\071\241s\b\000\000\003\377a\377\221 \000\000\003\377a\377\221(\000\000\003\377a\377\224(\000\000\000\004\020\000\000\000\000\000\003\377 \006\237\200\000\000\003\377 \006\237X\000\000\003\377 \000\000\000\000\000\000\000\000\000\000\002\000\000\002\252\071\312i "..., m_used = 0}}, m_user_space = 0x0, m_commit_lsn = 0, m_freed_space = 0x0, m_freed_pages = 0x0} __PRETTY_FUNCTION__ = "bool row_purge_poss_sec(purge_node_t*, dict_index_t*, const dtuple_t*, btr_pcur_t*, mtr_t*, bool)" #9 0x000002aa399a38ec in row_purge_remove_sec_if_poss_leaf (node=0x2aa3c7f9a28, index=0x3ff20069de8, entry=0x3ff38004da8) at /home/buildbot/s390x-ubuntu-2004-debug/build/storage/innobase/row/row0purge.cc:470 mtr = {m_start = true, m_commit = false, m_last = 0x0, m_last_offset = 0, m_log_mode = 0, m_modifications = 0, m_made_dirty = 0, m_inside_ibuf = 0, m_trim_pages = 0, m_user_space_id = 7, m_memo = {m_heap = 0x0, m_list = {<ilist<mtr_buf_t::block_t, void>> = {sentinel_ = {next = 0x3ff61ff96f0, prev = 0x3ff61ff96f0}}, size_ = 1}, m_size = 48, m_first_block = {<ilist_node<void>> = {next = 0x3ff61ff96d0, prev = 0x3ff61ff96d0}, m_buf_end = 0, m_magic_n = 375767, m_data = '\000' <repeats 11 times>, " ", '\000' <repeats 15 times>, "\001\000\000\000\000\000\000\003\377z\201\355\230\000\000\000\002\000\000\000\000\000\000\000\000\000\000\004\200\000\000\003\377a\377\231\324\000\000\000\000\000\000\004\230\000\000\003\377a\377\233\000\000\000\003\377\000\000\000\002\000\000\002\252<\202RK\000\000\003\377a\377\231\310\000\000\003\377a\377\233\000?\360\000\000\000\000\000\000\000\000\003\377\206=\345\000\000\000\003\377a\377\230\360\000\000\003\377 \b\216 \000\000\002\252<@\207\220\000\000\003\377a\377\235\350\000\000\000\000\000\000\000\005\000\000\003\377a\377\271\000\000\000\002\252\071za0\000\000\003\377{)h\263\000\000\000\000\001e7\v"..., m_used = 48}}, m_log = {m_heap = 0x0, m_list = {<ilist<mtr_buf_t::block_t, void>> = {sentinel_ = {next = 0x3ff61ff9930, prev = 0x3ff61ff9930}}, size_ = 1}, m_size = 0, m_first_block = {<ilist_node<void>> = {next = 0x3ff61ff9910, prev = 0x3ff61ff9910}, m_buf_end = 0, m_magic_n = 375767, m_data = "\000\000\003\377\200\237\304\267\000\000\003\377a\377\257\340\000\000\002\252<e\351\360\000\000\003\377\070\000M\340\000\000\003\377a\377\231\030\000\000\002\252:\225O\200\000\000\002\252:.\206\300\000\000\002\252\071\305y\224\000\000\003\377a\377\231\030\000\000\002\252\071`\213\374", '\000' <repeats 23 times>, " \000\000\003\377a\377\232\340\000\000\003\377a\377\232\340\000\000\003\377\070\000M\340\000\000\003\377\000\000\000\004\000\000\003\377a\377\231\200\000\000\002\252:\225O\200\000\000\003\377a\377\271\000\000\000\002\252\071\232\243\226\000\000\003\377a\377\231\200\000\000\003\377a\377\231\210\000\000\000\000\000\000\000\001\000\000\003\377\200\237\304\267"..., m_used = 0}}, m_user_space = 0x3ff2006a508, m_commit_lsn = 0, m_freed_space = 0x0, m_freed_pages = 0x0} pcur = {btr_cur = {index = 0x3ff20069de8, page_cur = {index = 0x0, rec = 0x3ff7be22a16 "", offsets = 0x0, block = 0x3ff7a81ed98}, purge_node = 0x2aa3c7f9a28, left_block = 0x0, thr = 0x0, flag = BTR_CUR_BINARY, tree_height = 1, up_match = 1, up_bytes = 0, low_match = 2, low_bytes = 0, n_fields = 0, n_bytes = 0, fold = 0, path_arr = 0x0, rtr_info = 0x3ff380051c8}, latch_mode = 2, old_stored = false, old_rec = 0x0, old_n_core_fields = 65535, old_n_fields = 0, rel_pos = 0, block_when_stored = {m_block = 0x0, m_page_id = {m_id = 0}}, modify_clock = 0, pos_state = BTR_PCUR_IS_POSITIONED, search_mode = PAGE_CUR_RTREE_LOCATE, trx_if_known = 0x0, old_rec_buf = 0x0, buf_size = 0} mode = BTR_MODIFY_LEAF search_result = ROW_FOUND success = true __PRETTY_FUNCTION__ = "bool row_purge_remove_sec_if_poss_leaf(purge_node_t*, dict_index_t*, const dtuple_t*)" #10 0x000002aa399a3e6e in row_purge_remove_sec_if_poss (node=0x2aa3c7f9a28, index=0x3ff20069de8, entry=0x3ff38004da8) at /home/buildbot/s390x-ubuntu-2004-debug/build/storage/innobase/row/row0purge.cc:571 success = 2930182691504 n_tries = 0 #11 0x000002aa399a4166 in row_purge_del_mark (node=0x2aa3c7f9a28) at /home/buildbot/s390x-ubuntu-2004-debug/build/storage/innobase/row/row0purge.cc:641 entry = 0x3ff38004da8 heap = 0x3ff38004d28 result = 128 #12 0x000002aa399a6c76 in row_purge_record_func (node=0x2aa3c7f9a28, undo_rec=0x2aa3c8251f8 "", thr=0x2aa3c7f9848, updated_extern=false) at /home/buildbot/s390x-ubuntu-2004-debug/build/storage/innobase/row/row0purge.cc:1192 clust_index = 0x3ff20236e88 purged = true __PRETTY_FUNCTION__ = "bool row_purge_record_func(purge_node_t*, trx_undo_rec_t*, const que_thr_t*, bool)" #13 0x000002aa399a6faa in row_purge (node=0x2aa3c7f9a28, undo_rec=0x2aa3c8251f8 "", thr=0x2aa3c7f9848) at /home/buildbot/s390x-ubuntu-2004-debug/build/storage/innobase/row/row0purge.cc:1253 purged = 232 updated_extern = false #14 0x000002aa399a71c2 in row_purge_step (thr=0x2aa3c7f9848) at /home/buildbot/s390x-ubuntu-2004-debug/build/storage/innobase/row/row0purge.cc:1302 purge_rec = {undo_rec = 0x2aa3c8251f8 "", roll_ptr = 14073748858931804} node = 0x2aa3c7f9a28 #15 0x000002aa398e12b6 in que_thr_step (thr=0x2aa3c7f9848) at /home/buildbot/s390x-ubuntu-2004-debug/build/storage/innobase/que/que0que.cc:865 node = 0x2aa3c7f9a28 old_thr = 0x2aa3c7f9848 trx = 0x3ff80a5c030 type = 13 __PRETTY_FUNCTION__ = "que_thr_t* que_thr_step(que_thr_t*)" #16 0x000002aa398e164c in que_run_threads_low (thr=0x2aa3c7f9848) at /home/buildbot/s390x-ubuntu-2004-debug/build/storage/innobase/que/que0que.cc:927 trx = 0x3ff80a5c030 next_thr = 0x2aa3c7f9848 __PRETTY_FUNCTION__ = "void que_run_threads_low(que_thr_t*)" #17 0x000002aa398e198a in que_run_threads (thr=0x2aa3c7f9848) at /home/buildbot/s390x-ubuntu-2004-debug/build/storage/innobase/que/que0que.cc:967 __PRETTY_FUNCTION__ = "void que_run_threads(que_thr_t*)" #18 0x000002aa399fe8f2 in srv_task_execute () at /home/buildbot/s390x-ubuntu-2004-debug/build/storage/innobase/srv/srv0srv.cc:1746 thr = 0x2aa3c7f9848 __PRETTY_FUNCTION__ = "bool srv_task_execute()" #19 0x000002aa399ff4ca in purge_worker_callback () at /home/buildbot/s390x-ubuntu-2004-debug/build/storage/innobase/srv/srv0srv.cc:1928 __PRETTY_FUNCTION__ = "void purge_worker_callback(void*)" ctx = 0x3ff38000b60 thd = 0x2aa3c8a5d28 #20 0x000002aa39cb0b6e in tpool::task_group::execute (this=0x2aa3b4828f8 <purge_task_group>, t=0x2aa3b4826c0 <purge_worker_task>) at /home/buildbot/s390x-ubuntu-2004-debug/build/tpool/task_group.cc:55 lk = {_M_device = 0x2aa3b482928 <purge_task_group+48>, _M_owns = false} #21 0x000002aa39cb10a6 in tpool::task::execute (this=0x2aa3b4826c0 <purge_worker_task>) at /home/buildbot/s390x-ubuntu-2004-debug/build/tpool/task.cc:47 No locals. #22 0x000002aa39ca6a06 in tpool::thread_pool_generic::worker_main (this=0x2aa3c65e9f0, thread_var=0x2aa3c65f720) at /home/buildbot/s390x-ubuntu-2004-debug/build/tpool/tpool_generic.cc:599 task = 0x2aa3b4826c0 <purge_worker_task> #23 0x000002aa39cb08ce in std::__invoke_impl<void, void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> (__f=@0x3ff5c000bd8: (void (tpool::thread_pool_generic::*)(tpool::thread_pool_generic * const, tpool::worker_data *)) 0x2aa39ca6920 <tpool::thread_pool_generic::worker_main(tpool::worker_data*)>, __t=@0x3ff5c000bd0: 0x2aa3c65e9f0) at /usr/include/c++/9/bits/invoke.h:73 No locals. #24 0x000002aa39cb06ae in std::__invoke<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> (__fn=@0x3ff5c000bd8: (void (tpool::thread_pool_generic::*)(tpool::thread_pool_generic * const, tpool::worker_data *)) 0x2aa39ca6920 <tpool::thread_pool_generic::worker_main(tpool::worker_data*)>) at /usr/include/c++/9/bits/invoke.h:95 No locals. #25 0x000002aa39cb0542 in std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> >::_M_invoke<0ul, 1ul, 2ul> (this=0x3ff5c000bc8) at /usr/include/c++/9/thread:244 No locals. #26 0x000002aa39cb04a4 in std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> >::operator() (this=0x3ff5c000bc8) at /usr/include/c++/9/thread:251 No locals. #27 0x000002aa39cb0472 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> > >::_M_run (this=0x3ff5c000bc0) at /usr/include/c++/9/thread:195 No locals. #28 0x000003ff8706a096 in ?? () from /lib/s390x-linux-gnu/libstdc++.so.6 No symbol table info available. #29 0x000003ff87287e66 in start_thread () from /lib/s390x-linux-gnu/libpthread.so.0 No symbol table info available. #30 0x000003ff86cfcd46 in ?? () from /lib/s390x-linux-gnu/libc.so.6 No symbol table info available. Thread 11 (Thread 0x3ff8767f900 (LWP 179409)): #0 0x000003ff8728ebbe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/s390x-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000002aa39d6cf00 in safe_cond_timedwait (cond=0x2aa3ab62a00 <thread_cache>, mp=0x2aa3ab62a70 <thread_cache+112>, abstime=0x3ff8767eb68, file=0x2aa39f0b1aa "/home/buildbot/s390x-ubuntu-2004-debug/build/include/mysql/psi/mysql_thread.h", line=1259) at /home/buildbot/s390x-ubuntu-2004-debug/build/mysys/thr_mutex.c:541 error = -2016252024 save_state = {global = {__data = {__lock = 1, __count = 0, __owner = 179409, __nusers = 1, __kind = 2, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = "\000\000\000\001\000\000\000\000\000\002\274\321\000\000\000\001\000\000\000\002", '\000' <repeats 19 times>, __align = 4294967296}, mutex = {__data = {__lock = 1, __count = 0, __owner = 179409, __nusers = 1, __kind = 3, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = "\000\000\000\001\000\000\000\000\000\002\274\321\000\000\000\001\000\000\000\003", '\000' <repeats 19 times>, __align = 4294967296}, file = 0x2aa39f0b88c "/home/buildbot/s390x-ubuntu-2004-debug/build/sql/thread_cache.h", name = 0x2aa39e86aad "LOCK_thread_cache", line = 166, count = 0, create_flags = 0, active_flags = 0, id = 193, thread = 4396023281920, locked_mutex = 0x2aa3ccd3c88, used_mutex = 0x2aa3ccd3d00, prev = 0x0, next = 0x0} #2 0x000002aa38fbaec0 in inline_mysql_cond_timedwait (that=0x2aa3ab62a00 <thread_cache>, mutex=0x2aa3ab62a70 <thread_cache+112>, abstime=0x3ff8767eb68, src_file=0x2aa39f0b88c "/home/buildbot/s390x-ubuntu-2004-debug/build/sql/thread_cache.h", src_line=176) at /home/buildbot/s390x-ubuntu-2004-debug/build/include/mysql/psi/mysql_thread.h:1259 locker = 0x0 state = {m_flags = 682, m_operation = (unknown: 956019406), m_cond = 0x3ff8767ea00, m_mutex = 0x3ff8767ea28, m_thread = 0x3ff8767eb68, m_timer_start = 1720697147282684, m_timer = 0x668fc13b, m_wait = 0x2aa3ab62b20 <thread_cache+288>} result = -2016252024 #3 0x000002aa38fbb1b8 in Thread_cache::park (this=0x2aa3ab62a00 <thread_cache>) at /home/buildbot/s390x-ubuntu-2004-debug/build/sql/thread_cache.h:176 error = 536874312 abstime = {tv_sec = 1720697447, tv_nsec = 282684000} connect = 0x0 flushed = false #4 0x000002aa38fba4d2 in do_handle_one_connection (connect=0x2aa3ccf8d28, put_in_cache=true) at /home/buildbot/s390x-ubuntu-2004-debug/build/sql/sql_connect.cc:1430 create_user = true thr_create_utime = 8415344960068 thd = 0x3ff20000d48 #5 0x000002aa38fba048 in handle_one_connection (arg=0x2aa3cccb528) at /home/buildbot/s390x-ubuntu-2004-debug/build/sql/sql_connect.cc:1319 connect = 0x2aa3cccb528 #6 0x000002aa3960b71e in pfs_spawn_thread (arg=0x2aa3cd05e58) at /home/buildbot/s390x-ubuntu-2004-debug/build/storage/perfschema/pfs.cc:2201 typed_arg = 0x2aa3cd05e58 user_arg = 0x2aa3cccb528 user_start_routine = 0x2aa38fb9fd0 <handle_one_connection(void*)> pfs = 0x3ff863bbf00 klass = 0x2aa3c422100 #7 0x000003ff87287e66 in start_thread () from /lib/s390x-linux-gnu/libpthread.so.0 No symbol table info available. #8 0x000003ff86cfcd46 in ?? () from /lib/s390x-linux-gnu/libc.so.6 No symbol table info available. Thread 10 (Thread 0x3ff78cdc900 (LWP 195526)): #0 0x000003ff8728ebbe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/s390x-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000002aa39ca83ea in __gthread_cond_timedwait (__cond=0x2aa3c65f020, __mutex=0x2aa3c65eb08, __abs_timeout=0x3ff78cdb5b8) at /usr/include/s390x-linux-gnu/c++/9/bits/gthr-default.h:872 No locals. #2 0x000002aa39cac910 in std::condition_variable::__wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x2aa3c65f020, __lock=@0x3ff78cdb8e8: {_M_device = 0x2aa3c65eb08, _M_owns = true}, __atime=@0x3ff78cdb698: {__d = {__r = 1720697213456983152}}) at /usr/include/c++/9/condition_variable:188 __s = {__d = {__r = 1720697213}} __ns = {__r = 456983152} __ts = {tv_sec = 1720697213, tv_nsec = 456983152} #3 0x000002aa39cab46a in std::condition_variable::wait_until<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x2aa3c65f020, __lock=@0x3ff78cdb8e8: {_M_device = 0x2aa3c65eb08, _M_owns = true}, __atime=@0x3ff78cdb770: {__d = {__r = 8416348170550248}}) at /usr/include/c++/9/condition_variable:121 __c_entry = {__d = {__r = 8416288170550560}} __s_entry = {__d = {__r = 1720697153456983464}} __delta = {__r = 59999999688} __s_atime = {__d = {__r = 1720697213456983152}} #4 0x000002aa39ca9e3c in std::condition_variable::wait_for<long, std::ratio<1l, 1000l> > (this=0x2aa3c65f020, __lock=@0x3ff78cdb8e8: {_M_device = 0x2aa3c65eb08, _M_owns = true}, __rtime=@0x2aa3c65eb30: {__r = 60000}) at /usr/include/c++/9/condition_variable:152 __reltime = {__r = 60000000000} #5 0x000002aa39ca63b2 in tpool::thread_pool_generic::wait_for_tasks (this=0x2aa3c65e9f0, lk=@0x3ff78cdb8e8: {_M_device = 0x2aa3c65eb08, _M_owns = true}, thread_data=0x2aa3c65f020) at /home/buildbot/s390x-ubuntu-2004-debug/build/tpool/tpool_generic.cc:499 __PRETTY_FUNCTION__ = "bool tpool::thread_pool_generic::wait_for_tasks(std::unique_lock<std::mutex>&, tpool::worker_data*)" #6 0x000002aa39ca6650 in tpool::thread_pool_generic::get_task (this=0x2aa3c65e9f0, thread_var=0x2aa3c65f020, t=0x3ff78cdb9a8) at /home/buildbot/s390x-ubuntu-2004-debug/build/tpool/tpool_generic.cc:552 lk = {_M_device = 0x2aa3c65eb08, _M_owns = true} __PRETTY_FUNCTION__ = "bool tpool::thread_pool_generic::get_task(tpool::worker_data*, tpool::task**)" #7 0x000002aa39ca69b6 in tpool::thread_pool_generic::worker_main (this=0x2aa3c65e9f0, thread_var=0x2aa3c65f020) at /home/buildbot/s390x-ubuntu-2004-debug/build/tpool/tpool_generic.cc:597 task = 0x2aa3b4826c0 <purge_worker_task> #8 0x000002aa39cb08ce in std::__invoke_impl<void, void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> (__f=@0x3ff201c03e8: (void (tpool::thread_pool_generic::*)(tpool::thread_pool_generic * const, tpool::worker_data *)) 0x2aa39ca6920 <tpool::thread_pool_generic::worker_main(tpool::worker_data*)>, __t=@0x3ff201c03e0: 0x2aa3c65e9f0) at /usr/include/c++/9/bits/invoke.h:73 No locals. #9 0x000002aa39cb06ae in std::__invoke<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> (__fn=@0x3ff201c03e8: (void (tpool::thread_pool_generic::*)(tpool::thread_pool_generic * const, tpool::worker_data *)) 0x2aa39ca6920 <tpool::thread_pool_generic::worker_main(tpool::worker_data*)>) at /usr/include/c++/9/bits/invoke.h:95 No locals. #10 0x000002aa39cb0542 in std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> >::_M_invoke<0ul, 1ul, 2ul> (this=0x3ff201c03d8) at /usr/include/c++/9/thread:244 No locals. #11 0x000002aa39cb04a4 in std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> >::operator() (this=0x3ff201c03d8) at /usr/include/c++/9/thread:251 No locals. #12 0x000002aa39cb0472 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> > >::_M_run (this=0x3ff201c03d0) at /usr/include/c++/9/thread:195 No locals. #13 0x000003ff8706a096 in ?? () from /lib/s390x-linux-gnu/libstdc++.so.6 No symbol table info available. #14 0x000003ff87287e66 in start_thread () from /lib/s390x-linux-gnu/libpthread.so.0 No symbol table info available. #15 0x000003ff86cfcd46 in ?? () from /lib/s390x-linux-gnu/libc.so.6 No symbol table info available. Thread 9 (Thread 0x3ff8165d900 (LWP 179327)): #0 0x000003ff8728ebbe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/s390x-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000002aa39d6cf00 in safe_cond_timedwait (cond=0x2aa3b3a3028 <COND_checkpoint>, mp=0x2aa3b3a2f78 <LOCK_checkpoint>, abstime=0x3ff8165cbf0, file=0x2aa3a10c610 "/home/buildbot/s390x-ubuntu-2004-debug/build/include/mysql/psi/mysql_thread.h", line=1259) at /home/buildbot/s390x-ubuntu-2004-debug/build/mysys/thr_mutex.c:541 error = -2016252024 save_state = {global = {__data = {__lock = 1, __count = 0, __owner = 179327, __nusers = 1, __kind = 2, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = "\000\000\000\001\000\000\000\000\000\002\274\177\000\000\000\001\000\000\000\002", '\000' <repeats 19 times>, __align = 4294967296}, mutex = {__data = {__lock = 1, __count = 0, __owner = 179327, __nusers = 1, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = "\000\000\000\001\000\000\000\000\000\002\274\177\000\000\000\001", '\000' <repeats 23 times>, __align = 4294967296}, file = 0x2aa3a10c65e "/home/buildbot/s390x-ubuntu-2004-debug/build/storage/maria/ma_servicethread.c", name = 0x2aa3a10c6ac "control->LOCK_control", line = 99, count = 0, create_flags = 0, active_flags = 0, id = 17, thread = 4395922479360, locked_mutex = 0x3ff74000f88, used_mutex = 0x3ff74001000, prev = 0x0, next = 0x0} #2 0x000002aa39547e22 in inline_mysql_cond_timedwait (that=0x2aa3b3a3028 <COND_checkpoint>, mutex=0x2aa3b3a2f78 <LOCK_checkpoint>, abstime=0x3ff8165cbf0, src_file=0x2aa3a10c65e "/home/buildbot/s390x-ubuntu-2004-debug/build/storage/maria/ma_servicethread.c", src_line=115) at /home/buildbot/s390x-ubuntu-2004-debug/build/include/mysql/psi/mysql_thread.h:1259 locker = 0x3ff8165caf0 state = {m_flags = 7, m_operation = (unknown: 2170931960), m_cond = 0x2aa3c532f00, m_mutex = 0x3ff879c5100, m_thread = 0x3ff863a3300, m_timer_start = 16095993603730558673, m_timer = 0x2aa3964eeb8 <my_timer_cycles()>, m_wait = 0x3ff863a3c28} result = -2124035480 #3 0x000002aa395482fc in my_service_thread_sleep (control=0x2aa3a993730 <checkpoint_control>, sleep_time=30000000000) at /home/buildbot/s390x-ubuntu-2004-debug/build/storage/maria/ma_servicethread.c:115 abstime = {tv_sec = 1720697168, tv_nsec = 416486000} res = 0 '\000' #4 0x000002aa3953975c in ma_checkpoint_background (arg=0x1e) at /home/buildbot/s390x-ubuntu-2004-debug/build/storage/maria/ma_checkpoint.c:725 interval = 30 sleeps = 930 sleep_time = 30 log_horizon_at_last_checkpoint = 4295005134 pagecache_flushes_at_last_checkpoint = 0 pages_bunch_size = 0 filter_param = {up_to_lsn = 4295005090, max_pages = 0} dfile = 0x0 kfile = 0x0 __PRETTY_FUNCTION__ = "ma_checkpoint_background" #5 0x000002aa3960b71e in pfs_spawn_thread (arg=0x2aa3c643828) at /home/buildbot/s390x-ubuntu-2004-debug/build/storage/perfschema/pfs.cc:2201 typed_arg = 0x2aa3c643828 user_arg = 0x1e user_start_routine = 0x2aa39539378 <ma_checkpoint_background> pfs = 0x3ff863a3300 klass = 0x2aa3c423000 #6 0x000003ff87287e66 in start_thread () from /lib/s390x-linux-gnu/libpthread.so.0 No symbol table info available. #7 0x000003ff86cfcd46 in ?? () from /lib/s390x-linux-gnu/libc.so.6 No symbol table info available. Thread 8 (Thread 0x3ff877ff900 (LWP 179386)): #0 0x000003ff8728e820 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/s390x-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000002aa39d6cbae in safe_cond_wait (cond=0x2aa3ab6bdd0 <COND_manager>, mp=0x2aa3ab6bd20 <LOCK_manager>, file=0x2aa39ec3760 "/home/buildbot/s390x-ubuntu-2004-debug/build/include/mysql/psi/mysql_thread.h", line=1222) at /home/buildbot/s390x-ubuntu-2004-debug/build/mysys/thr_mutex.c:487 error = -2021658656 save_state = {global = {__data = {__lock = 1, __count = 0, __owner = 179386, __nusers = 1, __kind = 2, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = "\000\000\000\001\000\000\000\000\000\002\274\272\000\000\000\001\000\000\000\002", '\000' <repeats 19 times>, __align = 4294967296}, mutex = {__data = {__lock = 1, __count = 0, __owner = 179386, __nusers = 1, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = "\000\000\000\001\000\000\000\000\000\002\274\272\000\000\000\001", '\000' <repeats 23 times>, __align = 4294967296}, file = 0x2aa39ec3d88 "/home/buildbot/s390x-ubuntu-2004-debug/build/sql/sql_manager.cc", name = 0x2aa39ec3e5d "LOCK_manager", line = 123, count = 0, create_flags = 0, active_flags = 0, id = 57, thread = 4396024854784, locked_mutex = 0x2aa3cbc9428, used_mutex = 0x2aa3cbc94a0, prev = 0x0, next = 0x0} #2 0x000002aa38dbb3d4 in inline_mysql_cond_wait (that=0x2aa3ab6bdd0 <COND_manager>, mutex=0x2aa3ab6bd20 <LOCK_manager>, src_file=0x2aa39ec3d88 "/home/buildbot/s390x-ubuntu-2004-debug/build/sql/sql_manager.cc", src_line=109) at /home/buildbot/s390x-ubuntu-2004-debug/build/include/mysql/psi/mysql_thread.h:1222 locker = 0x3ff877febf0 state = {m_flags = 7, m_operation = (unknown: 2273307640), m_cond = 0x2aa3c536100, m_mutex = 0x3ff879e1000, m_thread = 0x3ff863b9300, m_timer_start = 16095989800682939928, m_timer = 0x2aa3964eeb8 <my_timer_cycles()>, m_wait = 0x3ff863b9c28} result = 985054496 #3 0x000002aa38dbbac6 in handle_manager (arg=0x0) at /home/buildbot/s390x-ubuntu-2004-debug/build/sql/sql_manager.cc:109 cb = 0x0 error = 0 abstime = {tv_sec = 488, tv_nsec = 488} reset_flush_time = true __PRETTY_FUNCTION__ = "void* handle_manager(void*)" #4 0x000002aa3960b71e in pfs_spawn_thread (arg=0x2aa3c82ab98) at /home/buildbot/s390x-ubuntu-2004-debug/build/storage/perfschema/pfs.cc:2201 typed_arg = 0x2aa3c82ab98 user_arg = 0x0 user_start_routine = 0x2aa38dbb878 <handle_manager(void*)> pfs = 0x3ff863b9300 klass = 0x2aa3c421f00 #5 0x000003ff87287e66 in start_thread () from /lib/s390x-linux-gnu/libpthread.so.0 No symbol table info available. #6 0x000003ff86cfcd46 in ?? () from /lib/s390x-linux-gnu/libc.so.6 No symbol table info available. Thread 7 (Thread 0x3ff6ffff900 (LWP 179367)): #0 0x000003ff8728e820 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/s390x-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000003ff87063c08 in std::condition_variable::wait(std::unique_lock<std::mutex>&) () from /lib/s390x-linux-gnu/libstdc++.so.6 No symbol table info available. #2 0x000002aa39cb1330 in tpool::waitable_task::wait (this=0x2aa3b4826c0 <purge_worker_task>, lk=@0x3ff6fffe3f0: {_M_device = 0x2aa3b4826e0 <purge_worker_task+32>, _M_owns = true}) at /home/buildbot/s390x-ubuntu-2004-debug/build/tpool/task.cc:79 No locals. #3 0x000002aa39cb13a4 in tpool::waitable_task::wait (this=0x2aa3b4826c0 <purge_worker_task>) at /home/buildbot/s390x-ubuntu-2004-debug/build/tpool/task.cc:85 lk = {_M_device = 0x2aa3b4826e0 <purge_worker_task+32>, _M_owns = true} #4 0x000002aa39a3eb82 in trx_purge_wait_for_workers_to_complete () at /home/buildbot/s390x-ubuntu-2004-debug/build/storage/innobase/trx/trx0purge.cc:1235 notify_wait = true __PRETTY_FUNCTION__ = "void trx_purge_wait_for_workers_to_complete()" #5 0x000002aa39a3ee38 in trx_purge (n_tasks=3) at /home/buildbot/s390x-ubuntu-2004-debug/build/storage/innobase/trx/trx0purge.cc:1281 thr = 0x2aa3c7fa568 n_pages_handled = 9 __PRETTY_FUNCTION__ = "ulint trx_purge(ulint)" #6 0x000002aa399fedea in srv_do_purge (n_total_purged=0x3ff6fffe6d0) at /home/buildbot/s390x-ubuntu-2004-debug/build/storage/innobase/srv/srv0srv.cc:1840 n_pages_purged = 11 count = 16 n_use_threads = 3 rseg_history_len = 9 old_activity_count = 54 n_threads = 4 __PRETTY_FUNCTION__ = "size_t srv_do_purge(ulint*)" #7 0x000002aa399ff644 in purge_coordinator_callback_low () at /home/buildbot/s390x-ubuntu-2004-debug/build/storage/innobase/srv/srv0srv.cc:1945 sigcount = 1 woken_during_purge = 170 n_total_purged = 49 #8 0x000002aa399ff7ce in purge_coordinator_callback () at /home/buildbot/s390x-ubuntu-2004-debug/build/storage/innobase/srv/srv0srv.cc:1974 ctx = 0x3ff64000b60 thd = 0x2aa3c8af9e8 #9 0x000002aa39cb0b6e in tpool::task_group::execute (this=0x2aa3b482988 <purge_coordinator_task_group>, t=0x2aa3b482a18 <purge_coordinator_task>) at /home/buildbot/s390x-ubuntu-2004-debug/build/tpool/task_group.cc:55 lk = {_M_device = 0x2aa3b4829b8 <purge_coordinator_task_group+48>, _M_owns = false} #10 0x000002aa39cb10a6 in tpool::task::execute (this=0x2aa3b482a18 <purge_coordinator_task>) at /home/buildbot/s390x-ubuntu-2004-debug/build/tpool/task.cc:47 No locals. #11 0x000002aa39ca6a06 in tpool::thread_pool_generic::worker_main (this=0x2aa3c65e9f0, thread_var=0x2aa3c65ee20) at /home/buildbot/s390x-ubuntu-2004-debug/build/tpool/tpool_generic.cc:599 task = 0x2aa3b482a18 <purge_coordinator_task> #12 0x000002aa39cb08ce in std::__invoke_impl<void, void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> (__f=@0x2aa3c87ed68: (void (tpool::thread_pool_generic::*)(tpool::thread_pool_generic * const, tpool::worker_data *)) 0x2aa39ca6920 <tpool::thread_pool_generic::worker_main(tpool::worker_data*)>, __t=@0x2aa3c87ed60: 0x2aa3c65e9f0) at /usr/include/c++/9/bits/invoke.h:73 No locals. #13 0x000002aa39cb06ae in std::__invoke<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> (__fn=@0x2aa3c87ed68: (void (tpool::thread_pool_generic::*)(tpool::thread_pool_generic * const, tpool::worker_data *)) 0x2aa39ca6920 <tpool::thread_pool_generic::worker_main(tpool::worker_data*)>) at /usr/include/c++/9/bits/invoke.h:95 No locals. #14 0x000002aa39cb0542 in std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> >::_M_invoke<0ul, 1ul, 2ul> (this=0x2aa3c87ed58) at /usr/include/c++/9/thread:244 No locals. #15 0x000002aa39cb04a4 in std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> >::operator() (this=0x2aa3c87ed58) at /usr/include/c++/9/thread:251 No locals. #16 0x000002aa39cb0472 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> > >::_M_run (this=0x2aa3c87ed50) at /usr/include/c++/9/thread:195 No locals. #17 0x000003ff8706a096 in ?? () from /lib/s390x-linux-gnu/libstdc++.so.6 No symbol table info available. #18 0x000003ff87287e66 in start_thread () from /lib/s390x-linux-gnu/libpthread.so.0 No symbol table info available. #19 0x000003ff86cfcd46 in ?? () from /lib/s390x-linux-gnu/libc.so.6 No symbol table info available. Thread 6 (Thread 0x3ff6f7fe900 (LWP 202701)): #0 0x000003ff8728ebbe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/s390x-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000002aa39ca83ea in __gthread_cond_timedwait (__cond=0x2aa3c65f320, __mutex=0x2aa3c65eb08, __abs_timeout=0x3ff6f7fd5b8) at /usr/include/s390x-linux-gnu/c++/9/bits/gthr-default.h:872 No locals. #2 0x000002aa39cac910 in std::condition_variable::__wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x2aa3c65f320, __lock=@0x3ff6f7fd8e8: {_M_device = 0x2aa3c65eb08, _M_owns = true}, __atime=@0x3ff6f7fd698: {__d = {__r = 1720697205627018297}}) at /usr/include/c++/9/condition_variable:188 __s = {__d = {__r = 1720697205}} __ns = {__r = 627018297} __ts = {tv_sec = 1720697205, tv_nsec = 627018297} #3 0x000002aa39cab46a in std::condition_variable::wait_until<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x2aa3c65f320, __lock=@0x3ff6f7fd8e8: {_M_device = 0x2aa3c65eb08, _M_owns = true}, __atime=@0x3ff6f7fd770: {__d = {__r = 8416340340585386}}) at /usr/include/c++/9/condition_variable:121 __c_entry = {__d = {__r = 8416280340585533}} __s_entry = {__d = {__r = 1720697145627018444}} __delta = {__r = 59999999853} __s_atime = {__d = {__r = 1720697205627018297}} #4 0x000002aa39ca9e3c in std::condition_variable::wait_for<long, std::ratio<1l, 1000l> > (this=0x2aa3c65f320, __lock=@0x3ff6f7fd8e8: {_M_device = 0x2aa3c65eb08, _M_owns = true}, __rtime=@0x2aa3c65eb30: {__r = 60000}) at /usr/include/c++/9/condition_variable:152 __reltime = {__r = 60000000000} #5 0x000002aa39ca63b2 in tpool::thread_pool_generic::wait_for_tasks (this=0x2aa3c65e9f0, lk=@0x3ff6f7fd8e8: {_M_device = 0x2aa3c65eb08, _M_owns = true}, thread_data=0x2aa3c65f320) at /home/buildbot/s390x-ubuntu-2004-debug/build/tpool/tpool_generic.cc:499 __PRETTY_FUNCTION__ = "bool tpool::thread_pool_generic::wait_for_tasks(std::unique_lock<std::mutex>&, tpool::worker_data*)" #6 0x000002aa39ca6650 in tpool::thread_pool_generic::get_task (this=0x2aa3c65e9f0, thread_var=0x2aa3c65f320, t=0x3ff6f7fd9a8) at /home/buildbot/s390x-ubuntu-2004-debug/build/tpool/tpool_generic.cc:552 lk = {_M_device = 0x2aa3c65eb08, _M_owns = true} __PRETTY_FUNCTION__ = "bool tpool::thread_pool_generic::get_task(tpool::worker_data*, tpool::task**)" #7 0x000002aa39ca69b6 in tpool::thread_pool_generic::worker_main (this=0x2aa3c65e9f0, thread_var=0x2aa3c65f320) at /home/buildbot/s390x-ubuntu-2004-debug/build/tpool/tpool_generic.cc:597 task = 0x2aa3c892340 #8 0x000002aa39cb08ce in std::__invoke_impl<void, void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> (__f=@0x3ff5c000ba8: (void (tpool::thread_pool_generic::*)(tpool::thread_pool_generic * const, tpool::worker_data *)) 0x2aa39ca6920 <tpool::thread_pool_generic::worker_main(tpool::worker_data*)>, __t=@0x3ff5c000ba0: 0x2aa3c65e9f0) at /usr/include/c++/9/bits/invoke.h:73 No locals. #9 0x000002aa39cb06ae in std::__invoke<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> (__fn=@0x3ff5c000ba8: (void (tpool::thread_pool_generic::*)(tpool::thread_pool_generic * const, tpool::worker_data *)) 0x2aa39ca6920 <tpool::thread_pool_generic::worker_main(tpool::worker_data*)>) at /usr/include/c++/9/bits/invoke.h:95 No locals. #10 0x000002aa39cb0542 in std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> >::_M_invoke<0ul, 1ul, 2ul> (this=0x3ff5c000b98) at /usr/include/c++/9/thread:244 No locals. #11 0x000002aa39cb04a4 in std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> >::operator() (this=0x3ff5c000b98) at /usr/include/c++/9/thread:251 No locals. #12 0x000002aa39cb0472 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> > >::_M_run (this=0x3ff5c000b90) at /usr/include/c++/9/thread:195 No locals. #13 0x000003ff8706a096 in ?? () from /lib/s390x-linux-gnu/libstdc++.so.6 No symbol table info available. #14 0x000003ff87287e66 in start_thread () from /lib/s390x-linux-gnu/libpthread.so.0 No symbol table info available. #15 0x000003ff86cfcd46 in ?? () from /lib/s390x-linux-gnu/libc.so.6 No symbol table info available. Thread 5 (Thread 0x3ff87c6f900 (LWP 179323)): #0 0x000003ff8728ebbe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/s390x-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000002aa39d6cf00 in safe_cond_timedwait (cond=0x2aa3b4b2cf8 <COND_timer>, mp=0x2aa3b4b2c48 <LOCK_timer>, abstime=0x3ff87c6ecf8, file=0x2aa3a312830 "/home/buildbot/s390x-ubuntu-2004-debug/build/include/mysql/psi/mysql_thread.h", line=1259) at /home/buildbot/s390x-ubuntu-2004-debug/build/mysys/thr_mutex.c:541 error = -2016252024 save_state = {global = {__data = {__lock = 1, __count = 0, __owner = 179323, __nusers = 1, __kind = 2, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = "\000\000\000\001\000\000\000\000\000\002\274{\000\000\000\001\000\000\000\002", '\000' <repeats 19 times>, __align = 4294967296}, mutex = {__data = {__lock = 2, __count = 0, __owner = 179323, __nusers = 1, __kind = 3, __spins = 1, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = "\000\000\000\002\000\000\000\000\000\002\274{\000\000\000\001\000\000\000\003\000\001", '\000' <repeats 17 times>, __align = 8589934592}, file = 0x2aa3a312830 "/home/buildbot/s390x-ubuntu-2004-debug/build/include/mysql/psi/mysql_thread.h", name = 0x2aa3a3128bf "LOCK_timer", line = 1259, count = 0, create_flags = 0, active_flags = 0, id = 14, thread = 4396029507840, locked_mutex = 0x3ff7c0020d8, used_mutex = 0x3ff7c002150, prev = 0x0, next = 0x0} #2 0x000002aa39d6e19a in inline_mysql_cond_timedwait (that=0x2aa3b4b2cf8 <COND_timer>, mutex=0x2aa3b4b2c48 <LOCK_timer>, abstime=0x3ff87c6ecf8, src_file=0x2aa3a31287e "/home/buildbot/s390x-ubuntu-2004-debug/build/mysys/thr_timer.c", src_line=321) at /home/buildbot/s390x-ubuntu-2004-debug/build/include/mysql/psi/mysql_thread.h:1259 locker = 0x3ff87c6ebe8 state = {m_flags = 7, m_operation = (unknown: 2277960936), m_cond = 0x2aa3c531300, m_mutex = 0x3ff879c2800, m_thread = 0x3ff863a1d00, m_timer_start = 16095993664928975586, m_timer = 0x2aa3964eeb8 <my_timer_cycles()>, m_wait = 0x3ff863a2628} result = -2017006752 #3 0x000002aa39d6f03a in timer_handler (arg=0x0) at /home/buildbot/s390x-ubuntu-2004-debug/build/mysys/thr_timer.c:321 error = 110 top_time = 0x2aa3c65ec08 now = {tv_sec = 1720697153, tv_nsec = 357476000} abstime = {tv_sec = 1720697153, tv_nsec = 757486000} #4 0x000002aa3960b71e in pfs_spawn_thread (arg=0x2aa3c3f4608) at /home/buildbot/s390x-ubuntu-2004-debug/build/storage/perfschema/pfs.cc:2201 typed_arg = 0x2aa3c3f4608 user_arg = 0x0 user_start_routine = 0x2aa39d6ee98 <timer_handler> pfs = 0x3ff863a1d00 klass = 0x2aa3c422600 #5 0x000003ff87287e66 in start_thread () from /lib/s390x-linux-gnu/libpthread.so.0 No symbol table info available. #6 0x000003ff86cfcd46 in ?? () from /lib/s390x-linux-gnu/libc.so.6 No symbol table info available. Thread 4 (Thread 0x3ff876ff900 (LWP 179401)): #0 0x000003ff86c42614 in sigtimedwait () from /lib/s390x-linux-gnu/libc.so.6 No symbol table info available. #1 0x000002aa38c21be0 in my_sigwait (set=0x3ff876feb70, sig=0x3ff876feb5c, code=0x3ff876feb60) at /home/buildbot/s390x-ubuntu-2004-debug/build/include/my_pthread.h:193 siginfo = {si_signo = 1023, si_errno = -2022707232, si_code = 1023, __pad0 = -350768434, _sifields = {_pad = {1023, -350767560, 1023, -2022708752, 1023, -2016252024, 1023, -2022704896, 682, 952246428, 1023, -2022708752, 682, 952248982, 1023, -2022708736, 7, 22, 1023, -2019819776, 1023, 3017, 682, 971541092, 682, 985027408, 16538566, -845618776}, _kill = {si_pid = 1023, si_uid = 3944199736}, _timer = {si_tid = 1023, si_overrun = -350767560, si_sigval = {sival_int = 1023, sival_ptr = 0x3ff876fe9f0}}, _rt = {si_pid = 1023, si_uid = 3944199736, si_sigval = {sival_int = 1023, sival_ptr = 0x3ff876fe9f0}}, _sigchld = {si_pid = 1023, si_uid = 3944199736, si_status = 1023, si_utime = 4396030259080, si_stime = 4396023806208}, _sigfault = {si_addr = 0x3ffeb17b638, si_addr_lsb = 0, _bounds = {_addr_bnd = {_lower = 0x3ff87d26f88, _upper = 0x3ff876ff900}, _pkey = 1023}}, _sigpoll = {si_band = 4397695743544, si_fd = 1023}, _sigsys = {_call_addr = 0x3ffeb17b638, _syscall = 1023, _arch = 2272258544}}} #2 0x000002aa38c29130 in signal_hand (arg=0x0) at /home/buildbot/s390x-ubuntu-2004-debug/build/sql/mysqld.cc:3025 error = -2016252024 origin = 1023 set = {__val = {548869, 0 <repeats 15 times>}} sig = 0 __PRETTY_FUNCTION__ = "void* signal_hand(void*)" #3 0x000002aa3960b71e in pfs_spawn_thread (arg=0x2aa3c645198) at /home/buildbot/s390x-ubuntu-2004-debug/build/storage/perfschema/pfs.cc:2201 typed_arg = 0x2aa3c645198 user_arg = 0x0 user_start_routine = 0x2aa38c28f68 <signal_hand(void*)> pfs = 0x3ff863ba900 klass = 0x2aa3c422200 #4 0x000003ff87287e66 in start_thread () from /lib/s390x-linux-gnu/libpthread.so.0 No symbol table info available. #5 0x000003ff86cfcd46 in ?? () from /lib/s390x-linux-gnu/libc.so.6 No symbol table info available. Thread 3 (Thread 0x3ff809ff900 (LWP 179330)): #0 0x000003ff86cf8182 in syscall () from /lib/s390x-linux-gnu/libc.so.6 No symbol table info available. #1 0x000002aa39cb183e in my_getevents (ctx=0x3ff80a00000, min_nr=1, nr=256, ev=0x3ff809fc9c0) at /home/buildbot/s390x-ubuntu-2004-debug/build/tpool/aio_linux.cc:64 saved_errno = 0 ret = 12 #2 0x000002aa39cb1b34 in tpool::aio_linux::getevent_thread_routine (aio=0x2aa3c6b6490) at /home/buildbot/s390x-ubuntu-2004-debug/build/tpool/aio_linux.cc:108 ret = 12 MAX_EVENTS = 256 events = {{data = 0x0, obj = 0x2aa3c6f20b0, res = 65536, res2 = 0}, {data = 0x0, obj = 0x2aa3c6f1f30, res = 65536, res2 = 0}, {data = 0x0, obj = 0x2aa3c6f2170, res = 65536, res2 = 0}, {data = 0x0, obj = 0x2aa3c6f1ff0, res = 65536, res2 = 0}, {data = 0x0, obj = 0x2aa3c6f2230, res = 65536, res2 = 0}, {data = 0x0, obj = 0x2aa3c6f22f0, res = 65536, res2 = 0}, {data = 0x0, obj = 0x2aa3c6f2e30, res = 65536, res2 = 0}, {data = 0x0, obj = 0x2aa3c6f3070, res = 65536, res2 = 0}, {data = 0x0, obj = 0x2aa3c6f2ef0, res = 65536, res2 = 0}, {data = 0x0, obj = 0x2aa3c6f2d70, res = 65536, res2 = 0}, {data = 0x0, obj = 0x2aa3c6f2cb0, res = 65536, res2 = 0}, {data = 0x0, obj = 0x2aa3c6f2bf0, res = 65536, res2 = 0}, {data = 0x0, obj = 0x2aa3c6f2b30, res = 65536, res2 = 0}, {data = 0x0, obj = 0x2aa3c6f2bf0, res = 65536, res2 = 0}, {data = 0x0, obj = 0x2aa3c6f2cb0, res = 65536, res2 = 0}, {data = 0x0, obj = 0x2aa3c6f2d70, res = 65536, res2 = 0}, {data = 0x0, obj = 0x2aa3c6f2ef0, res = 65536, res2 = 0}, {data = 0x0, obj = 0x2aa3c6f3070, res = 65536, res2 = 0}, {data = 0x0, obj = 0x2aa3c6f2e30, res = 65536, res2 = 0}, {data = 0x0, obj = 0x2aa3c6f22f0, res = 65536, res2 = 0}, {data = 0x0, obj = 0x2aa3c6f2230, res = 65536, res2 = 0}, {data = 0x0, obj = 0x2aa3c6f2170, res = 65536, res2 = 0}, {data = 0x0, obj = 0x2aa3c6f20b0, res = 65536, res2 = 0}, {data = 0x0, obj = 0x2aa3c6f1ff0, res = 65536, res2 = 0}, {data = 0x0, obj = 0x2aa3c6f1f30, res = 65536, res2 = 0}, {data = 0x0, obj = 0x2aa3c6f07f0, res = 65536, res2 = 0}, {data = 0x0, obj = 0x2aa3c6f0730, res = 65536, res2 = 0}, {data = 0x0, obj = 0x2aa3c6f1630, res = 65536, res2 = 0}, {data = 0x0, obj = 0x2aa3c6f16f0, res = 65536, res2 = 0}, {data = 0x0, obj = 0x2aa3c6f17b0, res = 65536, res2 = 0}, {data = 0x0, obj = 0x2aa3c6f1870, res = 65536, res2 = 0}, {data = 0x0, obj = 0x2aa3c6f1930, res = 65536, res2 = 0}, {data = 0x0, obj = 0x2aa3c6f19f0, res = 65536, res2 = 0}, {data = 0x0, obj = 0x2aa3c6f1ab0, res = 65536, res2 = 0}, {data = 0x0, obj = 0x2aa3c6f1b70, res = 65536, res2 = 0}, {data = 0x0, obj = 0x2aa3c6f1c30, res = 65536, res2 = 0}, {data = 0x0, obj = 0x2aa3c6f1cf0, res = 65536, res2 = 0}, {data = 0x0, obj = 0x2aa3c6f1db0, res = 65536, res2 = 0}, {data = 0x0, obj = 0x2aa3c6f1e70, res = 65536, res2 = 0}, {data = 0x0, obj = 0x2aa3c6f1f30, res = 65536, res2 = 0}, {data = 0x0, obj = 0x2aa3c6f1ff0, res = 65536, res2 = 0}, {data = 0x0, obj = 0x2aa3c6f20b0, res = 65536, res2 = 0}, {data = 0x0, obj = 0x2aa3c6f2170, res = 65536, res2 = 0}, {data = 0x0, obj = 0x2aa3c6f2230, res = 65536, res2 = 0}, {data = 0x0, obj = 0x2aa3c6f22f0, res = 65536, res2 = 0}, {data = 0x0, obj = 0x2aa3c6f23b0, res = 65536, res2 = 0}, {data = 0x0, obj = 0x2aa3c6f2470, res = 65536, res2 = 0}, {data = 0x0, obj = 0x2aa3c6f2530, res = 65536, res2 = 0}, {data = 0x0, obj = 0x2aa3c6f25f0, res = 65536, res2 = 0}, {data = 0x0, obj = 0x2aa3c6f26b0, res = 65536, res2 = 0}, {data = 0x0, obj = 0x2aa3c6f2770, res = 65536, res2 = 0}, {data = 0x0, obj = 0x2aa3c6f2830, res = 65536, res2 = 0}, {data = 0x0, obj = 0x2aa3c6f28f0, res = 65536, res2 = 0}, {data = 0x0, obj = 0x2aa3c6f29b0, res = 65536, res2 = 0}, {data = 0x0, obj = 0x2aa3c6f2a70, res = 65536, res2 = 0}, {data = 0x0, obj = 0x2aa3c6f2b30, res = 65536, res2 = 0}, {data = 0x0, obj = 0x2aa3c6f2bf0, res = 65536, res2 = 0}, {data = 0x0, obj = 0x2aa3c6f2cb0, res = 65536, res2 = 0}, {data = 0x0, obj = 0x2aa3c6f2d70, res = 65536, res2 = 0}, {data = 0x0, obj = 0x2aa3c6f2e30, res = 65536, res2 = 0}, {data = 0x0, obj = 0x2aa3c6f2ef0, res = 65536, res2 = 0}, {data = 0x0, obj = 0x2aa3c6f2fb0, res = 65536, res2 = 0}, {data = 0x0, obj = 0x2aa3c6f3070, res = 65536, res2 = 0}, {data = 0x0, obj = 0x2aa3c6f3130, res = 65536, res2 = 0}, {data = 0x0, obj = 0x0, res = 0, res2 = 0} <repeats 182 times>, {data = 0x0, obj = 0x2aa3c6d6dd8, res = 0, res2 = 0}, {data = 0x0, obj = 0x0, res = 0, res2 = 0}, {data = 0x0, obj = 0x0, res = 0, res2 = 0}, {data = 0x0, obj = 0x0, res = 0, res2 = 0}, {data = 0x0, obj = 0x0, res = 0, res2 = 0}, {data = 0x0, obj = 0x0, res = 2930181500376, res2 = 0}, {data = 0x0, obj = 0x0, res = 0, res2 = 0}, {data = 0x0, obj = 0x0, res = 0, res2 = 0}, {data = 0x0, obj = 0x0, res = 4395909507384, res2 = 4396030259080}, {data = 0x3ff809ff900, obj = 0x2aa39cb2cca <std::__get_helper<1ul, tpool::aio_linux*>(std::_Tuple_impl<1ul, tpool::aio_linux*>&)+34>, res = 4395909507384, res2 = 2930181500376}} #3 0x000002aa39cb2c62 in std::__invoke_impl<void, void (*)(tpool::aio_linux*), tpool::aio_linux*> (__f=@0x2aa3c6d6de0: 0x2aa39cb1ad8 <tpool::aio_linux::getevent_thread_routine(tpool::aio_linux*)>) at /usr/include/c++/9/bits/invoke.h:60 No locals. #4 0x000002aa39cb2b24 in std::__invoke<void (*)(tpool::aio_linux*), tpool::aio_linux*> (__fn=@0x2aa3c6d6de0: 0x2aa39cb1ad8 <tpool::aio_linux::getevent_thread_routine(tpool::aio_linux*)>) at /usr/include/c++/9/bits/invoke.h:95 No locals. #5 0x000002aa39cb2a14 in std::thread::_Invoker<std::tuple<void (*)(tpool::aio_linux*), tpool::aio_linux*> >::_M_invoke<0ul, 1ul> (this=0x2aa3c6d6dd8) at /usr/include/c++/9/thread:244 No locals. #6 0x000002aa39cb299c in std::thread::_Invoker<std::tuple<void (*)(tpool::aio_linux*), tpool::aio_linux*> >::operator() (this=0x2aa3c6d6dd8) at /usr/include/c++/9/thread:251 No locals. #7 0x000002aa39cb296a in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (*)(tpool::aio_linux*), tpool::aio_linux*> > >::_M_run (this=0x2aa3c6d6dd0) at /usr/include/c++/9/thread:195 No locals. #8 0x000003ff8706a096 in ?? () from /lib/s390x-linux-gnu/libstdc++.so.6 No symbol table info available. #9 0x000003ff87287e66 in start_thread () from /lib/s390x-linux-gnu/libpthread.so.0 No symbol table info available. #10 0x000003ff86cfcd46 in ?? () from /lib/s390x-linux-gnu/libc.so.6 No symbol table info available. Thread 2 (Thread 0x3ff7a4ff900 (LWP 179331)): #0 0x000003ff8728e820 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/s390x-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000002aa39d6cbae in safe_cond_wait (cond=0x2aa3a9a5a00 <buf_pool+1280>, mp=0x2aa3a9a5900 <buf_pool+1024>, file=0x2aa3a2a1ad0 "/home/buildbot/s390x-ubuntu-2004-debug/build/storage/innobase/buf/buf0flu.cc", line=2315) at /home/buildbot/s390x-ubuntu-2004-debug/build/mysys/thr_mutex.c:487 error = 2052059104 save_state = {global = {__data = {__lock = 1, __count = 0, __owner = 179331, __nusers = 1, __kind = 2, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = "\000\000\000\001\000\000\000\000\000\002\274\203\000\000\000\001\000\000\000\002", '\000' <repeats 19 times>, __align = 4294967296}, mutex = {__data = {__lock = 1, __count = 0, __owner = 179331, __nusers = 1, __kind = 3, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = "\000\000\000\001\000\000\000\000\000\002\274\203\000\000\000\001\000\000\000\003", '\000' <repeats 19 times>, __align = 4294967296}, file = 0x2aa3a2a1ad0 "/home/buildbot/s390x-ubuntu-2004-debug/build/storage/innobase/buf/buf0flu.cc", name = 0x2aa3a298677 "flush_list_mutex", line = 2364, count = 0, create_flags = 0, active_flags = 0, id = 20, thread = 4395803605248, locked_mutex = 0x3ff70000f88, used_mutex = 0x3ff70001000, prev = 0x0, next = 0x0} #2 0x000002aa39b50770 in buf_flush_page_cleaner () at /home/buildbot/s390x-ubuntu-2004-debug/build/storage/innobase/buf/buf0flu.cc:2315 soft_lsn_limit = 0 oldest_lsn = 9986823 dirty_blocks = 15 dirty_pct = 3.9370078740157481 pct_lwm = 0 n_flushed = 21 n = 0 __PRETTY_FUNCTION__ = "void* buf_flush_page_cleaner(void*)" last_pages = 21 abstime = {tv_sec = 1720697028, tv_nsec = 381264000} lsn_limit = 0 last_activity_count = 0 #3 0x000003ff87287e66 in start_thread () from /lib/s390x-linux-gnu/libpthread.so.0 No symbol table info available. #4 0x000003ff86cfcd46 in ?? () from /lib/s390x-linux-gnu/libc.so.6 No symbol table info available. Thread 1 (Thread 0x3ff87d6b4d0 (LWP 179295)): #0 0x000003ff8729029e in pthread_kill () from /lib/s390x-linux-gnu/libpthread.so.0 No symbol table info available. #1 0x000002aa39d6613a in my_write_core (sig=6) at /home/buildbot/s390x-ubuntu-2004-debug/build/mysys/stacktrace.c:424 No locals. #2 0x000002aa39209be6 in handle_fatal_signal (sig=6) at /home/buildbot/s390x-ubuntu-2004-debug/build/sql/signal_handler.cc:357 curr_time = 1720697143 tm = {tm_sec = 43, tm_min = 25, tm_hour = 11, tm_mday = 11, tm_mon = 6, tm_year = 124, tm_wday = 4, tm_yday = 192, tm_isdst = 0, tm_gmtoff = 0, tm_zone = 0x2aa3c404e40 "UTC"} thd = 0x0 print_invalid_query_pointer = false #3 <signal handler called> No symbol table info available. #4 0x000003ff86cf1c92 in poll () from /lib/s390x-linux-gnu/libc.so.6 No symbol table info available. #5 0x000002aa38c302b0 in handle_connections_sockets () at /home/buildbot/s390x-ubuntu-2004-debug/build/sql/mysqld.cc:6305 sock = {fd = 21, m_psi = 0x2aa3cc05d00} error_count = 0 cAddr = {ss_family = 1, __ss_padding = "\003\377\353\027\272\300\000\000\000\000\000\000\000\000\000\000\003\377\353\027\306\017\000\000\002\252:\a\322\\\000\000\003\377\353\027\265\270\000\000\003\377\207\322o\210\000\000\003\377\353\027\265\310\000\000\003\377\207\322o\210\000\000\003\377\313\307\376{\000\000\002\252\070\302Br\000\000\003\377\353\027\265\310\000\000\000\a8\353\211\256\000\000\003\377\207\234\025\000\000\000\003\377\000\000\026p\000\000\002\252\071\350\212d", __ss_align = 2930139236964} retval = 1 socket_count = 2 fds = {{fd = 20, events = 1, revents = 0}, {fd = 21, events = 1, revents = 0}, {fd = 0, events = 9, revents = 24437}} pfs_fds = {{fd = 20, m_psi = 0x2aa3cc05b00}, {fd = 21, m_psi = 0x2aa3cc05d00}, {fd = 1023, m_psi = 0x4b64}} #6 0x000002aa38c2f7b2 in mysqld_main (argc=167, argv=0x2aa3c4058c0) at /home/buildbot/s390x-ubuntu-2004-debug/build/sql/mysqld.cc:5754 please_close_stdin = true ho_error = 0 new_thread_stack_size = 299008 user = 0x1 <error: Cannot access memory at address 0x1> #7 0x000002aa38c218bc in main (argc=26, argv=0x3ffeb17bad8) at /home/buildbot/s390x-ubuntu-2004-debug/build/sql/main.cc:25 No locals. Compressed file /home/buildbot/s390x-ubuntu-2004-debug/build/mysql-test/var/12/log/innodb_gis.rtree_purge-64k/mysqld.1/data/core - saving '/home/buildbot/s390x-ubuntu-2004-debug/build/mysql-test/var/12/log/innodb_gis.rtree_purge-64k/' to '/home/buildbot/s390x-ubuntu-2004-debug/build/mysql-test/var/log/innodb_gis.rtree_purge-64k/'

Marko Mäkelä added a comment - 2025-02-04 10:56 I found a new type of failure in this test: 10.6-MDEV-29445 b8d5620273afa5c7149058ac503f5625768ce22d (10.6 75b24a002ff5b95f2d6ae28231227c2588e7d8e9) innodb_gis.rtree_purge '32k' w22 [ fail ] Test ended at 2025-02-03 10:03:19   CURRENT_TEST: innodb_gis.rtree_purge mysqltest: At line 22: query 'delete from t' failed: <Unknown> (2013): Lost connection to server during query … mariadbd: /home/buildbot/amd64-ubuntu-2204-debug-ps/build/storage/innobase/buf/buf0buf.cc:2968: buf_block_t* buf_page_optimistic_fix(buf_block_t*, page_id_t): Assertion `state >= buf_page_t::UNFIXED' failed. … #8 0x00007f77ef1e7e96 in __assert_fail () from /lib/x86_64-linux-gnu/libc.so.6 #9 0x000055bf1a960853 in buf_page_optimistic_fix (block=0x7f77e1805bd0, id=...) at /home/buildbot/amd64-ubuntu-2204-debug-ps/build/storage/innobase/buf/buf0buf.cc:2968 #10 0x000055bf1a94044a in btr_pcur_optimistic_latch_leaves (pcur=0x7f77b8bcea80, latch_mode=0x7f77e8235ea4, mtr=0x7f77e8236940) at /home/buildbot/amd64-ubuntu-2204-debug-ps/build/storage/innobase/btr/btr0pcur.cc:230 #11 0x000055bf1a940b85 in btr_pcur_t::restore_position (this=0x7f77b8bcea80, restore_latch_mode=BTR_SEARCH_LEAF, mtr=0x7f77e8236940) at /home/buildbot/amd64-ubuntu-2204-debug-ps/build/storage/innobase/btr/btr0pcur.cc:358 #12 0x000055bf1a84d4a7 in sel_restore_position_for_mysql (same_user_rec=0x7f77e8236451, latch_mode=BTR_SEARCH_LEAF, pcur=0x7f77b8bcea80, moves_up=true, mtr=0x7f77e8236940) at /home/buildbot/amd64-ubuntu-2204-debug-ps/build/storage/innobase/row/row0sel.cc:3653 #13 0x000055bf1a850d0c in row_search_mvcc (buf=0x7f77b8ba3ec8 "\031", mode=PAGE_CUR_G, prebuilt=0x7f77b8bce8b8, match_mode=0, direction=1) at /home/buildbot/amd64-ubuntu-2204-debug-ps/build/storage/innobase/row/row0sel.cc:4742 #14 0x000055bf1a5ece44 in ha_innobase::general_fetch (this=0x7f77b8ba36c8, buf=0x7f77b8ba3ec8 "\031", direction=1, match_mode=0) at /home/buildbot/amd64-ubuntu-2204-debug-ps/build/storage/innobase/handler/ha_innodb.cc:9296 #15 0x000055bf1a5ed2e1 in ha_innobase::rnd_next (this=0x7f77b8ba36c8, buf=0x7f77b8ba3ec8 "\031") at /home/buildbot/amd64-ubuntu-2204-debug-ps/build/storage/innobase/handler/ha_innodb.cc:9499 The probable reason is inadequate locking or latching between purge and DML on SPATIAL INDEX . I assume that a recently finished purge batch had marked the R-tree page freed in the file. At the time of the assertion failure, no purge coordinator or worker task was active; only a seemingly idle purge_truncation_task occurs in any stack trace output.