[MDEV-14925] trx->rsegs.m_noredo.undo is corrupted during ROLLBACK Created: 2018-01-11 Updated: 2020-08-25 Resolved: 2018-03-08 |
|
| Status: | Closed |
| Project: | MariaDB Server |
| Component/s: | Storage Engine - InnoDB |
| Affects Version/s: | 10.2.9, 10.2.10, 10.2.12 |
| Fix Version/s: | N/A |
| Type: | Bug | Priority: | Major |
| Reporter: | Juan | Assignee: | Marko Mäkelä |
| Resolution: | Cannot Reproduce | Votes: | 0 |
| Labels: | innodb | ||
| Environment: |
RHEL 6.5 x64 on physical server. |
||
| Issue Links: |
|
||||||||
| Sprint: | 10.2.14 | ||||||||
| Description |
|
Problem started after an upgrade 10.0.22 -> 10.2.9 and continues with 10.2.12
|
| Comments |
| Comment by Marko Mäkelä [ 2018-03-08 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
The crash happens when executing a rollback statement. In InnoDB, all tables and records are exclusively locked and changes written upfront. Rollback should replay the undo log backwards, decrementing the trx->undo_no all the way to 0. Finally, the transaction with no net changes to tables will be committed. MariaDB 10.2 (and MySQL 5.7) introduced a separate tablespace for temporary tables and temporary undo logs. Until then, temporary tables were treated as persistent. It turns out that the function trx_roll_pop_top_rec_of_trx() is actually prioritizing the undo logs of persistent tables. Only after all persistent changes have been rolled back, InnoDB would move to processing any temporary undo log records that are past the rollback savepoint. While I have refactored this code in MariaDB 10.2 and 10.3, this logic of prioritizing persisted undo logs in rollback remains, just like it is in MySQL 5.7. The cause of the crash is a wrong page reference. In the core dump, I successfully looked up two undo pages: one in the persistent tablespace (system tablespace; innodb_undo_tablespaces was not used) and another in the temporary tablespace. The one in the temporary tablespace appears to be empty (containing only garbage after the start=end=0x110 byte offset). The one in the persistent tablespace is almost full with records, but there is a record at offset 0xb4c, not 0xb4b = 2891. Here is the wrong pointer:
The temporary tablespace does not have page number 376886, because it is much smaller:
In the 3 undo logs of the transaction, the last_page_no should be equal to top_page_no:
That is not the case for the temporary undo log:
Page 90544 exists in the temporary tablespace, but it is empty apart from an undo log header, which contains a MySQLXid that does not match the trx->xid. We also have trx->rsegs.m_noredo.undo.guess_block which matches much of the temporary undo log data structure in memory:
The guess_block contains only inserts to a single temporary table, in ascending order of a BIGINT AUTO_INCREMENT PRIMARY KEY column, with no gaps in the AUTO_INCREMENT values or the undo_no. The first undo record is at byte offset 0x38 and the last one at 0x2f8d. Each record is 21 bytes. The last undo_no is 0x88610 or 558,608. If this was the very last record written by the transaction, this would mean that there were 558,608 rows modified by the transaction. In addition to at least this one temporary table, the transaction modified 25 persistent tables (the size of trx->mod_tables). This temporary table had the PRIMARY KEY and one UNIQUE INDEX on 3 columns. The top_offset is pointing to a undo record in the middle of the guess_block. The guess_block should be the very last temporary undo log block that was written in the course of the transaction before rollback. But if this block were valid for the transaction, the top_undo_no should be much bigger (558,608 instead of 13). In trx->rsegs.m_redo.insert_undo.guess_block.frame the last undo_no is 0x88611, or 558,609 (one larger than the last written temporary undo log record).
It remains a mystery how and why exactly the trx->rsegs.m_noredo.undo was corrupted. Currently my best guess is that a stray write from some other thread corrupted the memory while the explicit ROLLBACK from the client connection was being executed. There are not many assignments to top_page_no, and the FIL_PAGE_OFFSET in the undo pages looks sensible. While painstakingly analyzing the core dump, I made a few observations that should eventually be addressed. But these should not explain the failure.
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Marko Mäkelä [ 2018-03-08 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Anyone who encounters or is able to repeat this, please reopen the bug and submit details. |