[MDEV-20543] Latching order violation during B-tree operations Created: 2019-09-10  Updated: 2019-11-22  Resolved: 2019-11-22

Status: Closed
Project: MariaDB Server
Component/s: Storage Engine - InnoDB
Affects Version/s: 10.2.2, 10.3.0, 10.4.0
Fix Version/s: N/A

Type: Bug Priority: Major
Reporter: Marko Mäkelä Assignee: Thirunarayanan Balathandayuthapani
Resolution: Not a Bug Votes: 0
Labels: upstream

Issue Links:
Blocks
blocks MDEV-19514 Defer change buffer merge until pages... Closed
Relates
relates to MDEV-14637 Latching order violation during btr_c... Closed

 Description   

It turns out that there are more causes to MDEV-14637 than the too small secondary index record size estimates that were fixed in MySQL 5.7.23.

With the following patch (extracted from MDEV-19514) applied to MySQL 5.7 or MariaDB Server 10.2 or later, the bug is repeatable:

diff --git a/storage/innobase/buf/buf0buf.cc b/storage/innobase/buf/buf0buf.cc
index 4f9ecf10ec7..ecd90053d68 100644
--- a/storage/innobase/buf/buf0buf.cc
+++ b/storage/innobase/buf/buf0buf.cc
@@ -4617,6 +4617,8 @@ got_block:
 		fix_block->made_dirty_with_no_latch = dirty_with_no_latch;
 	}
 
+	rw_lock_x_lock(&block->lock);
+	rw_lock_x_unlock(&block->lock);
 	mtr_memo_type_t	fix_type;
 
 	switch (rw_latch) {

./mtr --big-test --mysqld=--innodb-page-size=4k innodb.innodb_bulk_create_index

mysql-5.7.27

CURRENT_TEST: innodb.innodb_bulk_create_index
mysqltest: In included file ./suite/innodb/include/innodb_bulk_create_index.inc at line 84:
included from /mariadb/mysql-server/mysql-test/suite/innodb/t/innodb_bulk_create_index.test at line 41:
At line 84: query 'SELECT * FROM t1 WHERE title = 'a3000'' failed: 2013: Lost connection to MySQL server during query
Version: '5.7.27-debug-log'  socket: '/dev/shm/5.7/mysql-test/var/tmp/mysqld.1.sock'  port: 13000  Source distribution
########################################
DEADLOCK of threads detected!
rw-lock 0x7f188e748968 --Thread 139743545079552 has waited at buf0buf.cc line 4620 for 0  seconds the semaphore:
X-lock on RW-latch at 0x7f188e748968 created in file buf0buf.cc line 1460
a writer (thread id 139743127922432) has reserved it in mode  exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file row0row.cc line 1075
Last time write locked in file /mariadb/mysql-server/storage/innobase/include/mtr0mtr.ic line 153
Locked: thread 139743127922432 file mtr0mtr.ic line 153  X-LOCK
Locked: thread 139743127922432 file mtr0mtr.ic line 153  X-LOCK
rw-lock 0x7f188e72a030 --Thread 139743127922432 has waited at buf0buf.cc line 4620 for 0  seconds the semaphore:
X-lock (wait_ex) on RW-latch at 0x7f188e72a030 created in file buf0buf.cc line 1460
a writer (thread id 139743127922432) has reserved it in mode  wait exclusive
number of readers 1, waiters flag 0, lock_word: ffffffffefffffff
Last time read locked in file btr0cur.cc line 5736
Last time write locked in file /mariadb/mysql-server/storage/innobase/include/mtr0mtr.ic line 117
Locked: thread 139743545079552 file btr0cur.cc line 5736  S-LOCK
Locked: thread 139743545079552 file btr0cur.cc line 5736  S-LOCK
2019-09-10T07:31:49.289423Z 0 [ERROR] [FATAL] InnoDB: ######################################## Deadlock Detected!

My original work-around from MDEV-14637 (always acquire the index->lock in X mode for BTR_MODIFY_TREE operations) prevents the crash, but would introduce a very prominent performance regression.



 Comments   
Comment by Marko Mäkelä [ 2019-09-10 ]

I am afraid that we must address this bug before implementing MDEV-19514.

This bug causes 2 crashes in mysql-test-run when the MDEV-19514 changes are present:

  • innodb.innodb_defragment (a test that is specific to MariaDB)
  • innodb.innodb_bulk_create_index,4k (basically the same test as in MySQL 5.7)
Comment by Marko Mäkelä [ 2019-09-10 ]

This was also reported as MySQL Bug #96819.

Comment by Thirunarayanan Balathandayuthapani [ 2019-11-22 ]

It is a false alarm issue. We should do lock the page in buf_page_get_gen()

Generated at Thu Feb 08 09:00:17 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.