Marko Mäkelä added a comment - 2018-03-08 14:45 The crash happens when executing a rollback statement. In InnoDB, all tables and records are exclusively locked and changes written upfront. Rollback should replay the undo log backwards, decrementing the trx->undo_no all the way to 0. Finally, the transaction with no net changes to tables will be committed. MariaDB 10.2 (and MySQL 5.7) introduced a separate tablespace for temporary tables and temporary undo logs. Until then, temporary tables were treated as persistent. It turns out that the function trx_roll_pop_top_rec_of_trx() is actually prioritizing the undo logs of persistent tables. Only after all persistent changes have been rolled back, InnoDB would move to processing any temporary undo log records that are past the rollback savepoint. While I have refactored this code in MariaDB 10.2 and 10.3, this logic of prioritizing persisted undo logs in rollback remains, just like it is in MySQL 5.7. The cause of the crash is a wrong page reference. In the core dump, I successfully looked up two undo pages: one in the persistent tablespace (system tablespace; innodb_undo_tablespaces was not used) and another in the temporary tablespace. The one in the temporary tablespace appears to be empty (containing only garbage after the start=end=0x110 byte offset). The one in the persistent tablespace is almost full with records, but there is a record at offset 0xb4c, not 0xb4b = 2891. Here is the wrong pointer: (gdb) thread 1 (gdb) frame 13 (gdb) p *undo $1 = {id = 0, type = 1, state = 1, del_marks = 0, trx_id = 19350037632, xid = {formatID = 1, gtrid_length = 24, bqual_length = 0, data = "MySQLXid\346\001\000\000\000\000\000\000Y%\343\v", '\000' <repeats 107 times>}, dict_operation = 0, table_id = 6197768787689408977, rseg = 0x7f43b90c6360, space = 4294967294, hdr_page_no = 90544, hdr_offset = 86, last_page_no = 90544, size = 1, empty = 0, top_page_no = 376886, top_offset = 2891, top_undo_no = 13, guess_block = 0x7f3fed2ac008, withdraw_clock = 0, undo_list = {prev = 0x0, next = 0x0}} … The temporary tablespace does not have page number 376886, because it is much smaller: (gdb) frame 7 (gdb) p *space $3 = {name = 0x7f43b4a2b9e0 "innodb_temporary", id = 4294967294, max_lsn = 0, stop_ios = false, stop_new_ops = false, is_being_truncated = false, purpose = FIL_TYPE_TEMPORARY, chain = {count = 1, start = 0x7f43bcb24090, end = 0x7f43bcb24090, node = &fil_node_t::chain}, size = 98304, size_in_header = 98304, free_len = 950, free_limit = 95872, recv_size = 0, flags = 0, … In the 3 undo logs of the transaction, the last_page_no should be equal to top_page_no : (gdb) p trx->rsegs.m_redo.insert_undo.last_page_no $4 = 4653060 (gdb) p trx->rsegs.m_redo.insert_undo.top_page_no $5 = 4653060 (gdb) p trx->rsegs.m_redo.update_undo.last_page_no $6 = 3964969 (gdb) p trx->rsegs.m_redo.update_undo.top_page_no $7 = 3964969 That is not the case for the temporary undo log: (gdb) p trx->rsegs.m_noredo.undo.top_page_no $8 = 376886 (gdb) p trx->rsegs.m_noredo.undo.last_page_no $9 = 90544 Page 90544 exists in the temporary tablespace, but it is empty apart from an undo log header, which contains a MySQLXid that does not match the trx->xid . We also have trx->rsegs.m_noredo.undo.guess_block which matches much of the temporary undo log data structure in memory: (gdb) p *trx->rsegs.m_noredo.undo $10 = {id = 0, type = 1, state = 1, del_marks = 0, trx_id = 19350037632, xid = { formatID = 1, gtrid_length = 24, bqual_length = 0, data = "MySQLXid\346\001\000\000\000\000\000\000Y%\343\v", '\000' <repeats 107 times>}, dict_operation = 0, table_id = 6197768787689408977, rseg = 0x7f43b90c6360, space = 4294967294, hdr_page_no = 90544, hdr_offset = 86, last_page_no = 90544, size = 1, empty = 0, top_page_no = 376886, top_offset = 2891, top_undo_no = 13, guess_block = 0x7f3fed2ac008, withdraw_clock = 0, undo_list = {prev = 0x0, next = 0x0}} The guess_block contains only inserts to a single temporary table, in ascending order of a BIGINT AUTO_INCREMENT PRIMARY KEY column, with no gaps in the AUTO_INCREMENT values or the undo_no . The first undo record is at byte offset 0x38 and the last one at 0x2f8d. Each record is 21 bytes. The last undo_no is 0x88610 or 558,608. If this was the very last record written by the transaction, this would mean that there were 558,608 rows modified by the transaction. In addition to at least this one temporary table, the transaction modified 25 persistent tables (the size of trx->mod_tables ). This temporary table had the PRIMARY KEY and one UNIQUE INDEX on 3 columns. The top_offset is pointing to a undo record in the middle of the guess_block . The guess_block should be the very last temporary undo log block that was written in the course of the transaction before rollback. But if this block were valid for the transaction, the top_undo_no should be much bigger (558,608 instead of 13). In trx->rsegs.m_redo.insert_undo.guess_block.frame the last undo_no is 0x88611, or 558,609 (one larger than the last written temporary undo log record). In trx->rsegs.m_redo.update_undo.guess_block.frame the last undo_no is 0x88330 (much smaller than the first record in the last written temporary undo log page, 0x883cf). (gdb) p *trx->rsegs.m_redo.insert_undo $11 = {id = 1, type = 1, state = 1, del_marks = 0, trx_id = 19350037632, xid = { formatID = -1, gtrid_length = 0, bqual_length = 0, data = '\000' <repeats 127 times>}, dict_operation = 0, table_id = 228954336, rseg = 0x7f43b317e710, space = 0, hdr_page_no = 4653060, hdr_offset = 86, last_page_no = 4653060, size = 1, empty = 1, top_page_no = 4653060, top_offset = 272, top_undo_no = 0, guess_block = 0x7f39e4aa8240, withdraw_clock = 0, undo_list = {prev = 0x0, next = 0x0}} (gdb) p *trx->rsegs.m_redo.update_undo $12 = {id = 0, type = 2, state = 1, del_marks = 1, trx_id = 19350037632, xid = { formatID = 1, gtrid_length = 24, bqual_length = 0, data = "MySQLXid\346\001\000\000\000\000\000\000Y%\343\v", '\000' <repeats 107 times>}, dict_operation = 0, table_id = 4832885630348296191, rseg = 0x7f43b317e710, space = 0, hdr_page_no = 3964969, hdr_offset = 7083, last_page_no = 3964969, size = 1, empty = 1, top_page_no = 3964969, top_offset = 7269, top_undo_no = 323531, guess_block = 0x7f3d22574710, withdraw_clock = 0, undo_list = {prev = 0x0, next = 0x0}} It remains a mystery how and why exactly the trx->rsegs.m_noredo.undo was corrupted. Currently my best guess is that a stray write from some other thread corrupted the memory while the explicit ROLLBACK from the client connection was being executed. There are not many assignments to top_page_no , and the FIL_PAGE_OFFSET in the undo pages looks sensible. While painstakingly analyzing the core dump, I made a few observations that should eventually be addressed. But these should not explain the failure. It would be more logical to execute rollback in strict descending order of records written ( undo_no ), as the sequence is common to both persistent and temporary undo log records of a transaction. For temporary tablespaces, fsp_init_file_page_low() should zero out earlier contents. This would not only avoid confusion when analyzing core dumps, but also avoid leakage of data from encrypted (persistent) tablespaces to the temporary tablespace file. For temporary tablespaces, page checksums should be enabled in order to catch any file system corruption. (I do not think that this failure can be explained by any file system corruption.)

Marko Mäkelä added a comment - 2018-03-08 15:00 Anyone who encounters or is able to repeat this, please reopen the bug and submit details. The support customer who encountered the bug switched to ENGINE=Aria for temporary tables in order to prevent the problem from reoccurring.