[MDEV-23885] Crash in ha_delete_hash_node() in btr_search_drop_page_hash_index() Created: 2020-10-04 Updated: 2021-05-02 Resolved: 2021-03-05 |
|
| Status: | Closed |
| Project: | MariaDB Server |
| Component/s: | Storage Engine - InnoDB |
| Affects Version/s: | 10.4.13 |
| Fix Version/s: | N/A |
| Type: | Bug | Priority: | Major |
| Reporter: | Valerii Kravchuk | Assignee: | Juan |
| Resolution: | Incomplete | Votes: | 0 |
| Labels: | need_feedback | ||
| Issue Links: |
|
||||||||
| Description |
|
The following crash happened:
In the gdb backtrace from the core we see the following crashing thread:
|
| Comments |
| Comment by Marko Mäkelä [ 2020-10-05 ] | ||||||||||||
|
In 10.4.13, the identified source code line in btr_search_drop_page_hash_index() appears to be as follows:
I do not think that we should use ib_mutex_t for any rw_lock_t operations in release builds. thiru, please work with the customer and try to identify the mutex and also get a disassembly of both btr_search_drop_page_hash_index() and the code that it is calling. Since 10.4.13, we have made several changes to the code, including | ||||||||||||
| Comment by Marko Mäkelä [ 2020-10-05 ] | ||||||||||||
|
Sorry, I was mistakenly looking at btr0sea.cc:1472 (the caller was btr0btr.cc:1472). The line btr0sea.cc:1248 is the last line of the following statement:
It still remains to be resolved which mutex we would be acquiring here. In 10.5, this code was simplified as part of | ||||||||||||
| Comment by Thirunarayanan Balathandayuthapani [ 2020-10-05 ] | ||||||||||||
|
From stack trace, it looks like the first 2 frames are corrupted because line number is so big.
| ||||||||||||
| Comment by Marko Mäkelä [ 2020-10-06 ] | ||||||||||||
|
The crash occurs in ha_delete_hash_node() when the adaptive hash index entries for a page are being freed. Without installing the debug symbols (I’m not using a rpm-based GNU/Linux distribution), it seems that the crash occurs in the final mem_heap_free_top() call in the macro HASH_DELETE_AND_COMPACT():
Without having the debugging symbols, I am guessing that the inlined invocation of mem_heap_free_top() is attempting to compute the following:
The byte offset 8 could be heap->base.end. A few instructions later we are applying other offsets, such as 0x30 or 0x48, which probably correspond to mem_block_info_t::len and mem_block_info_t::free, which ought to be 0x18 bytes apart. The cause could be a hardware failure (the memory is spontaneously corrupting data) or a rogue write somewhere in the server process, possibly directly related to the adaptive hash index. We have a rather similar crash report in | ||||||||||||
| Comment by Juan [ 2020-10-06 ] | ||||||||||||
|
Here is the correct gdb output from the customer:
| ||||||||||||
| Comment by Marko Mäkelä [ 2020-10-07 ] | ||||||||||||
|
The offset 0x10 does not match my guess about the crash being located in the inlined mem_heap_free_top() invocation. My revised guess would be the following assignment in the macro HASH_DELETE_AND_COMPACT:
For the only expansion of this macro in ha_delete_hash_node(), this should correspond to the following:
The offset of 8 bytes matches the type declaration, because sizeof(ulint)==8 on this instruction set architecture:
My conclusion from my previous comment remains valid: such memory corruption could be caused by hardware failure or a hard-to-reproduce bug, possibly in the adaptive hash index. Some race conditions between buffer pool resizing and the adaptive hash index have been fixed since the 10.4.13 release. I would advise to test the memory subsystem (for example, sudo memtester … does not even require a reboot) or to disable the subsystem where the crash occurred:
| ||||||||||||
| Comment by Marko Mäkelä [ 2020-10-07 ] | ||||||||||||
|
Because I do not even know whether it would be possible to extract the value of the problematic pointer. It would seem to depend on whether the signal handler is saving the values of all registers, and whether the debugger would ‘restore’ the register values when traversing the stack frames. I know that GDB does that for the stack pointer and the frame pointer, so it might be possible. In any case, because the hash table for the adaptive hash index is likely allocated from the buffer pool, we would not be much wiser. We might know whether the cause was a null pointer, but not much more. We do know that the SIGSEGV was triggered at the following instruction:
Can we get the following output from GDB?
I am asking to dump this register in several stack frames, so that we can assess whether the register value in the interesting stack frame could have been properly saved and restored by the debugger. |