[MDEV-27025] insert-intention lock conflicts with waiting ORDINARY lock Created: 2021-11-11 Updated: 2023-06-27 Resolved: 2022-01-19 |
|
| Status: | Closed |
| Project: | MariaDB Server |
| Component/s: | Storage Engine - InnoDB |
| Affects Version/s: | 10.2, 10.3, 10.4, 10.5, 10.6, 10.7, 10.8 |
| Fix Version/s: | 10.5.14, 10.6.6, 10.7.2, 10.8.1, 10.3.35, 10.4.25 |
| Type: | Bug | Priority: | Critical |
| Reporter: | Vladislav Lesin | Assignee: | Vladislav Lesin |
| Resolution: | Fixed | Votes: | 1 |
| Labels: | None | ||
| Attachments: |
|
||||||||||||||||||||||||||||||||||||||||
| Issue Links: |
|
||||||||||||||||||||||||||||||||||||||||
| Description |
|
We have two transactions and one record. The first transaction holds ORDINARY S-lock on the record, the second transaction created waiting ORDINARY X-lock and waits for the first transaction. Then the first transaction requests insert-intention lock on the record. And this lock conflicts with the waiting ORDINARY X-lock of the second transaction. What causes deadlock. Why it should conflict? The first transaction already holds lock on the record. And the second's transaction lock contains "waiting" flag. Let's take a look 10.6 code:
Neither lock_rec_insert_check_and_lock() nor lock_rec_other_has_conflicting() doesn't check conflicting lock is in waiting state and it already waits for the requesting insert-intention lock transaction. The test is attached: ii-conflicts-waiting.test |
| Comments |
| Comment by Valerii Kravchuk [ 2021-11-11 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Matthias Leich [ 2021-12-14 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Vladislav Lesin [ 2021-12-28 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Some update. During fixing the above bugs I found out that the initial fix is wrong. We need to preserve the invariant that any lock in locks queue can wait only the lock which is located before the waiting lock in the queue. I added the corresponding code, but RQG testing showed one more crash, which I have not fixed yet. See commit message for details (https://github.com/MariaDB/server/tree/bb-10.6-MDEV-27025-deadlock). | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Vladislav Lesin [ 2021-12-30 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Update. I got rid of SEGFAULT mentioned the previous comment, but RQG testing still shows the case when there are two granted conflicting locks on the same record. It still happens during "XA prepare" when all S-locks of the XA are released and X-lock of the other transaction is granted because it is located before the conflicting XA X-lock in the queue despite that XA X-lock was created before the granted X-lock of the other transaction. So the question is how such order of locks in the queue is possible. Still debugging it. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Vladislav Lesin [ 2022-01-03 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Update. I have fixed the above error. Matthias created special RQG config file to cause the above bugs, local RQG testing of my fix with this file contains no above errors. I pushed my branch for buildbot testing and in parallel launched local RQG testing with InnoDB_standard.cc config. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Marko Mäkelä [ 2022-01-04 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
This is great work. I cannot see anything obviously wrong in the logic. I pushed some suggested cleanup. If you agree and tests pass, it should be eventually merged to the final commit. mleich, please test this extensively with RQG. Unfortunately, we had to relax a debug assertion in lock_rec_queue_validate() due to the improved conflict handling. Instead of asserting that the conflicting lock is exclusive, we assert that it must be at least shared. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Matthias Leich [ 2022-01-06 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
origin/bb-10.6- | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Sergei Golubchik [ 2022-01-10 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
I think it has to be backported. At least to 10.5 | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Vladislav Lesin [ 2022-01-10 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Update: I have started trx_lock_t::wait_trx backporting to 10.5. I have also ported some debug checks, some of them fail on mtr tests due to wrong trx_lock_t::wait_trx reset. 10.5 and 10.6 locking code differs. As an example, in 10.5 lock_rec_move_low() invokes lock_reset_lock_and_trx_wait() before lock_rec_add_to_queue(). lock_reset_lock_and_trx_wait() resets trx_lock_t::wait_trx, what causes assertion failure in lock_rec_add_to_queue(). 10.6 does not invoke lock_reset_lock_and_trx_wait() from lock_rec_move(), it just does necessary actions inline instead. So I need to catch all such errors. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Vladislav Lesin [ 2022-01-11 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Update: backported trx_lock_t::wait_trx and the fix to 10.5(branch bb-10.5- | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Vladislav Lesin [ 2022-01-12 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Update: did some code cleanup, fixed embedded server compilation error caused by my changes, did some code research to make sure my changes will not break Galera cluster and innodb-lock-schedule-algorithm=VATS,, passed the branch for RQG testing. My local RQG testing have not finished yet, but it looks promising, at least I don't see any crashes during several hours. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Matthias Leich [ 2022-01-13 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
origin/bb-10.5- | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Vladislav Lesin [ 2022-01-13 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
I squashed the commits and rebased the branches. In 10.6 branch I added some details in the comment to clarify why we don't need to lock lock_sys.wait_mutex to check trx->lock.wait_trx. marko, you have already reviewed 10.6 branch, could you please also review 10.5 branch? | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Vladislav Lesin [ 2022-01-13 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
bb-10.5- | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Marko Mäkelä [ 2022-01-14 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
The 10.6 version has already been reviewed and tested. That one is OK to push. Backporting to earlier versions is tricky and potentially risky, because the locking subsystem was heavily refactored in the 10.6 release. I posted some minor comments to the 10.5 version. I did not find anything really wrong there. I think that we’d better wait for additional test results from our customer before pushing the 10.5 version. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Marko Mäkelä [ 2022-03-08 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
This change was reverted, because it caused the incorrect-result bug | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Vladislav Lesin [ 2022-08-19 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
There is some interesting commit in MySQL-8: "Bug #11745929 CHANGE LOCK PRIORITY SO THAT THE TRANSACTION HOLDING S-LOCK GETS X-LOCK". It should be analyzed, it might fix the issue. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Vladislav Lesin [ 2022-10-26 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
I have analysed "Bug #11745929 CHANGE LOCK PRIORITY SO THAT THE TRANSACTION HOLDING S-LOCK GETS X-LOCK" commit from Oracle. The general reason why we reverted But, what if trx 1 holds S-lock, and tries to acquire X-lock on the same record after trx 2 created waiting X-lock? Why trx 1 should wait for trx 2? This is exactly the case of the above MySQL fix. More commonly it can be formulated as "insert-intention locks must not overtake a waiting ordinary or gap locks". I think it could be useful for us to port that commit to fix the above issue. Besides, it contains some useful optimization: `lock_rec_find_set_bit` which searches for the first bit set in a bitmap used bit-by-bit loop. Now it uses 13x times faster implementation which tries to skip 64,then 32,16, or 8 bits at a time. This is important for WAITING locks which have just a single bit set, in a bitmap with number of bits equal to the number of records on a page (which can be ~500). If we port it some time, we should also port "Bug #34123159 Assertion failure: lock0lock.cc:5161:lock_rec_has_expl(LOCK_X | LOCK_REC_NOT_GAP". One more note. In |