[MDEV-24863] InnoDB: Failing assertion: index == cursor->index in btr_search_update_hash_on_delete with innodb_evict_tables_on_commit_debug Created: 2021-02-14  Updated: 2023-06-08  Resolved: 2021-02-22

Status: Closed
Project: MariaDB Server
Component/s: Storage Engine - InnoDB
Affects Version/s: 10.5, 10.6
Fix Version/s: 10.2.38, 10.3.29, 10.4.19, 10.5.10, 10.6.0

Type: Bug Priority: Major
Reporter: Elena Stepanova Assignee: Thirunarayanan Balathandayuthapani
Resolution: Fixed Votes: 0
Labels: regression

Issue Links:
Duplicate
duplicates MDEV-24874 Draft: Assertion `index == cursor->in... Closed
is duplicated by MDEV-24982 heap-use-after-free in rec_get_offset... Closed
Problem/Incident
causes MDEV-27700 SUMMARY: AddressSanitizer: heap-use-a... Closed
Relates
relates to MDEV-22456 Dropping the adaptive hash index may ... Closed

 Description   

--source include/have_innodb.inc
--source include/have_sequence.inc
 
--let $cache= `select @@table_open_cache`
--let $adaptive= `select @@innodb_adaptive_hash_index`
--let $evict= `select @@innodb_evict_tables_on_commit_debug`
 
SET GLOBAL TABLE_OPEN_CACHE= 10,
           INNODB_ADAPTIVE_HASH_INDEX= 1,
           INNODB_EVICT_TABLES_ON_COMMIT_DEBUG= 1;
 
CREATE TABLE t1 (
  id INT AUTO_INCREMENT,
  a TIMESTAMP,
  b BINARY(132),
  PRIMARY KEY (a,b),
  KEY(id)
) ENGINE=InnoDB;
INSERT INTO t1 SELECT seq, '2012-12-12', CONCAT('s',seq) FROM seq_1_to_116;
 
CREATE TABLE t2 (pk INT PRIMARY KEY) ENGINE=InnoDB;
CREATE TABLE t3 (pk INT PRIMARY KEY) ENGINE=InnoDB;
CREATE TABLE t4 (pk INT PRIMARY KEY) ENGINE=InnoDB;
CREATE TABLE t5 (pk INT PRIMARY KEY) ENGINE=InnoDB;
CREATE TABLE t6 (pk INT PRIMARY KEY) ENGINE=InnoDB;
CREATE TABLE t7 (pk INT PRIMARY KEY) ENGINE=InnoDB;
CREATE TABLE t8 (pk INT PRIMARY KEY) ENGINE=InnoDB;
 
START TRANSACTION;
DELETE FROM t1;
INSERT INTO t2 VALUES (1);
INSERT INTO t3 VALUES (1);
INSERT INTO t4 VALUES (1);
INSERT INTO t5 VALUES (1);
INSERT INTO t6 VALUES (1);
INSERT INTO t7 VALUES (1);
INSERT INTO t8 VALUES (1);
COMMIT;
 
--source include/restart_mysqld.inc
 
# Cleanup
DROP TABLE t1, t2, t3, t4, t5, t6, t7, t8;
eval SET GLOBAL TABLE_OPEN_CACHE= $cache,
                INNODB_ADAPTIVE_HASH_INDEX= $adaptive,
                INNODB_EVICT_TABLES_ON_COMMIT_DEBUG= $evict;

10.5 b3df194e

2021-02-14 23:59:15 0x7f6d537fe700  InnoDB: Assertion failure in file /data/src/10.5/storage/innobase/btr/btr0sea.cc line 1805
InnoDB: Failing assertion: index == cursor->index
 
#6  0x000055afd612692a in ut_dbg_assertion_failed (expr=0x55afd68681b3 "index == cursor->index", file=0x55afd6867a48 "/data/src/10.5/storage/innobase/btr/btr0sea.cc", line=1805) at /data/src/10.5/storage/innobase/ut/ut0dbg.cc:60
#7  0x000055afd618a714 in btr_search_update_hash_on_delete (cursor=0x7f6d537fce90) at /data/src/10.5/storage/innobase/btr/btr0sea.cc:1805
#8  0x000055afd617022c in btr_cur_optimistic_delete_func (cursor=0x7f6d537fce90, flags=0, mtr=0x7f6d537fd130) at /data/src/10.5/storage/innobase/btr/btr0cur.cc:5557
#9  0x000055afd60730d0 in row_purge_remove_sec_if_poss_leaf (node=0x55afd845b748, index=0x7f6d44013d38, entry=0x7f6d44012ae8) at /data/src/10.5/storage/innobase/row/row0purge.cc:519
#10 0x000055afd60731f8 in row_purge_remove_sec_if_poss (node=0x55afd845b748, index=0x7f6d44013d38, entry=0x7f6d44012ae8) at /data/src/10.5/storage/innobase/row/row0purge.cc:567
#11 0x000055afd6073408 in row_purge_del_mark (node=0x55afd845b748) at /data/src/10.5/storage/innobase/row/row0purge.cc:637
#12 0x000055afd6074e3b in row_purge_record_func (node=0x55afd845b748, undo_rec=0x55afd846a038 "\001\065\016\002\024", thr=0x55afd845b568, updated_extern=false) at /data/src/10.5/storage/innobase/row/row0purge.cc:1047
#13 0x000055afd6075111 in row_purge (node=0x55afd845b748, undo_rec=0x55afd846a038 "\001\065\016\002\024", thr=0x55afd845b568) at /data/src/10.5/storage/innobase/row/row0purge.cc:1108
#14 0x000055afd6075291 in row_purge_step (thr=0x55afd845b568) at /data/src/10.5/storage/innobase/row/row0purge.cc:1157
#15 0x000055afd5fe1370 in que_thr_step (thr=0x55afd845b568) at /data/src/10.5/storage/innobase/que/que0que.cc:946
#16 0x000055afd5fe1608 in que_run_threads_low (thr=0x55afd845b568) at /data/src/10.5/storage/innobase/que/que0que.cc:1008
#17 0x000055afd5fe186e in que_run_threads (thr=0x55afd845b568) at /data/src/10.5/storage/innobase/que/que0que.cc:1048
#18 0x000055afd60baa26 in srv_task_execute () at /data/src/10.5/storage/innobase/srv/srv0srv.cc:1871
#19 0x000055afd60bb235 in purge_worker_callback () at /data/src/10.5/storage/innobase/srv/srv0srv.cc:2034
#20 0x000055afd62cc692 in tpool::task_group::execute (this=0x55afd7904940 <purge_task_group>, t=0x55afd79046c0 <purge_worker_task>) at /data/src/10.5/tpool/task_group.cc:55
#21 0x000055afd62cca0e in tpool::task::execute (this=0x55afd79046c0 <purge_worker_task>) at /data/src/10.5/tpool/task.cc:47
#22 0x000055afd62c4fd5 in tpool::thread_pool_generic::worker_main (this=0x55afd82a1bb0, thread_var=0x55afd82b17d0) at /data/src/10.5/tpool/tpool_generic.cc:546
#23 0x000055afd62cc49a in std::__invoke_impl<void, void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> (__f=@0x7f6d5c003508: (void (tpool::thread_pool_generic::*)(class tpool::thread_pool_generic * const, struct tpool::worker_data *)) 0x55afd62c4f3c <tpool::thread_pool_generic::worker_main(tpool::worker_data*)>, __t=@0x7f6d5c003500: 0x55afd82a1bb0) at /usr/include/c++/9/bits/invoke.h:73
#24 0x000055afd62cc37c in std::__invoke<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> (__fn=@0x7f6d5c003508: (void (tpool::thread_pool_generic::*)(class tpool::thread_pool_generic * const, struct tpool::worker_data *)) 0x55afd62c4f3c <tpool::thread_pool_generic::worker_main(tpool::worker_data*)>) at /usr/include/c++/9/bits/invoke.h:95
#25 0x000055afd62cc29b in std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> >::_M_invoke<0ul, 1ul, 2ul> (this=0x7f6d5c0034f8) at /usr/include/c++/9/thread:244
#26 0x000055afd62cc236 in std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> >::operator() (this=0x7f6d5c0034f8) at /usr/include/c++/9/thread:251
#27 0x000055afd62cc216 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> > >::_M_run (this=0x7f6d5c0034f0) at /usr/include/c++/9/thread:195
#28 0x00007f6d68248d84 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#29 0x00007f6d68364609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#30 0x00007f6d67f38293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Reproducible on 10.5, 10.6.
Couldn't reproduce on 10.4.
The test case is not applicable to non-debug builds due to the use of a debug variable.



 Comments   
Comment by Marko Mäkelä [ 2021-02-15 ]

This could be a real bug that was introduced in MDEV-22456. When the assertion fails in purge, we do have index->id == cursor->index->id, for the secondary index id on table test.t1. That is, when the evicted table definition was reloaded into the data dictionary cache by purge, we missed the opportunity to reuse a previously evicted dict_index_t (which we were unable to discard due to some adaptive hash index entries pointing to it).

Side note: the test can be simplified: there is no need to restore any global parameters that were temporarily changed before a restart.

For me, the crash occurs on purge before the server is restarted.

Comment by Elena Stepanova [ 2021-02-15 ]

> Side note: the test can be simplified: there is no need to restore any global parameters that were temporarily changed before a restart.

The reason for the redundancy is that the server restart is actually not needed there, the failure happens at some point soon after the COMMIT. I added the restart because I didn't like putting an arbitrary sleep there, but surely it can be done smarter than this.

Comment by Matthias Leich [ 2021-02-17 ]

Caused by latest experiences (rr with option "chaos" runs well again) some additional test campaign was started. 3000 tests with no new + bad effects.

Generated at Thu Feb 08 09:33:16 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.