All that happend after a system crash caused by a game (X:Rebirth). As the storage contains information about private and business e-mails I am not willing and legally not allowed to disclose the data.

__memcpy_ssse3 () at ../sysdeps/x86_64/multiarch/memcpy-ssse3.S:2848

2848    ../sysdeps/x86_64/multiarch/memcpy-ssse3.S: Datei oder Verzeichnis nicht gefunden.

(gdb) frame 4

#4  trx_purge_get_next_rec (n_pages_handled=n_pages_handled@entry=0x7fffdcc0dde0, heap=0x555557a4e8c0) at /usr/src/debug/mariadb-10.0.22/storage/xtradb/trx/trx0purge.cc:908

908     /usr/src/debug/mariadb-10.0.22/storage/xtradb/trx/trx0purge.cc: Datei oder Verzeichnis nicht gefunden.

(gdb) info locals

rec = 0x7fffe7f6d008 ""

rec2 = <optimized out>

offset = 4104

page_no = 633

space = 0

zip_size = 0

mtr = {memo = {heap = 0x0, used = 32, 

    data = "\001\000\000\000\000\000\000\000\200\310\377\342\377\177\000\000\001\000\000\000\000\000\000\000\200\321\377\342\377\177", '\000' <repeats 58 times>, "\t\000\000\000\000\000\000\000\200\000\000\000\000\000\000\000\260\000\000\000\000\000\000\000\002\000\000\000\000\000\000\000\t\000\000\000\062", '\000' <repeats 19 times>, "[\000\000\000n", '\000' <repeats 19 times>, "w\000\000\000|", '\000' <repeats 11 times>, " \000\000\330\377\177\000\000\200", '\000' <repeats 23 times>..., base = {count = 93825018611448, start = 0x68, end = 0x7fffd80266c8}, list = {prev = 0x555555dc3214 <dict_table_stats_lock(dict_table_t*, unsigned long)+596>, next = 0x11}}, log = {heap = 0x0, used = 0, 

    data = "\000\000\000\000\000\000\000\000]\000\000\000n\000\000\000\000\252\220y%\rg\220\000\000\000\000\000\000\000\000\000\t\000\330\377\177\000\000\300\350\244WUU\000\000\000\000\000\000\000\000\000\000\300\066\343VUU\000\000\000\000\000\000\000\000\000\000\350\343\244WUU\000\000_\031\316UUU\000\000\000\000\000\000\000\000\000\000\025", '\000' <repeats 16 times>, "\004", '\000' <repeats 15 times>, "\252\220y%\rg\220h\004\000\000\000\000\000\000\000\252\220y%\rg\220\200p\345VUU\000\000\300\350\244WUU\000\000(\351\244WUU\000\000\300\350\244WUU\000\000\300\066\343VUU\000\000<6\323UUU\000\000"..., base = {count = 0, start = 0x0, end = 0x555557a4e3e8}, list = {prev = 0x555557a22158, 

      next = 0x7ffff5e2e254 <__GI___libc_malloc+84>}}, inside_ibuf = 0, modifications = 0, made_dirty = 0, n_log_recs = 0, n_freed_pages = 0, log_mode = 21, start_lsn = 8104, end_lsn = 10405299918667295232, trx = 0x0}

(gdb) p *n_pages_handled

$1 = 166

Thanks, Matthias!

If there was a system crash that was resolved by forcibly switching the power off, it is possible that we are witnessing the results of another part of the hardware/firmware/software stack failing. Maybe fsync() is not working on the hardware as expected, and some writes are being reordered, causing a situation where some of the recovered InnoDB data pages correspond to a wrong logical time (too old or too new LSN).

Admittedly, it is unlikely that a Linux kernel hang triggered by a GPU intensive game would cause the hard disk firmware to hang. Perhaps the system was not completely frozen, and it was just in the middle of performing a write when the power was switched off?

How were the configuration parameters innodb_doublewrite, innodb_flush_method, innodb_checksum_algorithm, innodb_checksums set? Were any non-default InnoDB or XtraDB configuration parameters specified?

I did some more code review. It seems to me that purge should sequentially read the undo logs of each purgeable transaction. The wrong purge_sys->offset (4104 == 0x1008) should actually have been read from the same page, at some earlier address than 0x1008. The next-record pointer is in the first two bytes of the undo log record, and it must not be past the currently marked end of the page; see trx_undo_page_get_next_rec().
I think that the undo log pages are being written in an append-only fashion, so the next-record pointer should be in the immediately preceding record.

It seems that trx_undo_page_get_next_rec() can start reading an undo page from an arbitrary offset (purge_sys->hdr_offset) that we could have read as corrupted from some other page. There are quite a few bytes "0x10 0x08" in the undo log page, and of course it is also possible that purge_sys->hdr_offset was directly initialized as 0x1008 from some other page.

With the available data, it is not possible for me to debug this further. If you still have the data and are willing to investigate, it would be interesting to see which undo pages were accessed by trx0purge.cc before the corruption was noticed. That could then allow us to determine the cause of the corruption. Because I do not remember seeing anything like this while working on MySQL and InnoDB since 2003, I would tend to believe that the problem lies outside InnoDB. That said, I am not fully familiar of the changes made in XtraDB or MariaDB.

Hello Marko,

I attached the complete config of the server instance as it is returned by the --verbose --help option. I still have the data available. What futher information can I provide?

complete-config.txt

Hello Matthias,
Sorry, I missed your update.
I see that the InnoDB doublewrite buffer and the page checksums were enabled. So, crash recovery should have performed as expected. It is theoretically possible that a corrupted page accidentally got a valid-looking checksum, though.

The page checksum of the attached file undo_page.bin (which was extracted from gdb) is valid, which suggests that there was no recent redo log activity for the page, or a redo log checkpoint was written by one of the startup attempts before the code crashed when accessing the seemingly corrupted undo log.

As far as I can tell, given that you (quite understandably) do not want to share the data files for deeper analysis, the only way to move forward with this is that someone starts up the server on the data files under a debugger, setting breakpoints in InnoDB code to figure out where the corruption originally occurs.

Less likely options are that someone else runs into this bug, or that you are able to create a reduced test case for repeating the problem.

One more thing that you could perhaps try is to check the diagnostics of the storage medium. Does smartctl -a report any serious errors, such as sector relocation? I am thinking of a possible chain of events like this: The server crashed. On the first crash recovery attempt (applying the redo log), reading the undo log page failed, and the hard drive substituted an empty page (filled with zeros). This would pass the InnoDB page checksum. Then, the redo log records would be applied and the page would be rewritten, except for some ‘holes’ that were not covered by the redo log records.

Also, I wonder if there is some tool like memtest86 which would test the reliability of file systems and the block device. Running memtest86 or similar could also be a good idea, because InnoDB page checksums are only calculated immediately before writing a block back to disk. A low-probability RAM corruption caused by unreliable hardware could go unnoticed for a long time.

While using Thunderbird since this issue happened to my Kontact MariaDB database, I totally forgot about this issue report.

I checked the SMART values of my storage device and everything is fine with it. I also die an overnight memtest86 run to test my system memory, also without any issues.

To be able to use KDE's Kontact again, I deleted the old data (including the corrupted database) an redownloaded the emails from the IMAP server.

So I am sorry not being able to help to resolve this anymore. Maybe it was a weird combination of different circumstances that led tot this situation.

I think this can be closed as WONTFIX or something similar.

Thanks to all for your efforts.