[MDEV-23986] [ERROR] [FATAL] InnoDB: Page ... name ... page_type ... key_version 1 lsn ... compressed_len ... Created: 2020-10-19 Updated: 2020-10-21 Resolved: 2020-10-21 |
|
| Status: | Closed |
| Project: | MariaDB Server |
| Component/s: | Storage Engine - InnoDB |
| Affects Version/s: | 10.5 |
| Fix Version/s: | 10.5.7 |
| Type: | Bug | Priority: | Major |
| Reporter: | Matthias Leich | Assignee: | Marko Mäkelä |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | debug, page_compression, regression, rr-profile | ||
| Issue Links: |
|
||||||||||||||||||||||||||||
| Description |
|
|
| Comments |
| Comment by Matthias Leich [ 2020-10-19 ] | |||||||||||||||||||||||||||||||||||
|
| |||||||||||||||||||||||||||||||||||
| Comment by Marko Mäkelä [ 2020-10-19 ] | |||||||||||||||||||||||||||||||||||
|
thiru, please debug this further. I think that the problem is that we are compressing or decompressing the 64KiB page in the wrong way. The page appears to compress to 512 bytes. Well before the SIGKILL we had the following:
The $3 matches exactly the contents of *page@512 when recovery is reporting page corruption. We must find out what we are doing wrong in the compression or encryption. I suspect that the affected version is 10.4 (which introduced innodb_checksum_algorithm=full_crc32) or an even earlier version. | |||||||||||||||||||||||||||||||||||
| Comment by Marko Mäkelä [ 2020-10-20 ] | |||||||||||||||||||||||||||||||||||
|
I debugged this a bit further. The assertion fails because we apparently decrypt the 512 bytes wrong. In fil_encrypt_buf_for_full_crc32(), the bytes before encrypting are as follows:
It is a little curious that we have a sequence of 252 zero bytes at the end. The total length of the preceding block is 256 bytes, while the 252 NUL bytes and 4 CRC-32C bytes would make up another 256 bytes. Note that the length is encoded as mach_read_from_2(src_frame + FIL_PAGE_TYPE) == 0x8002, which will trip the assertion failure on recovery. Those bytes were intentionally written by fil_page_compress_for_full_crc32():
It turns out that since
I think that we must skip all the assertions for the full_crc32 format. There is a third one right after the decompress_with_slot: label. That one should fail for unencrypted page_compressed pages as well. Please try to recover the dataset after fixing the assertions. As far as I can tell, it should be easy to write a test case based on this analysis. It might be as simple as FLUSH TABLES…FOR EXPORT followed by ALTER TABLE…DISCARD TABLESPACE and ALTER TABLE…IMPORT TABLESPACE. | |||||||||||||||||||||||||||||||||||
| Comment by Marko Mäkelä [ 2020-10-20 ] | |||||||||||||||||||||||||||||||||||
|
It is possible that the changes in | |||||||||||||||||||||||||||||||||||
| Comment by Marko Mäkelä [ 2020-10-20 ] | |||||||||||||||||||||||||||||||||||
|
At the time of the SIGKILL, the latest checkpoint LSN is 153136497. When that checkpoint was written, we invoked log_write_checkpoint_info(end_lsn=154755265), which is what the start-up message displays. However, the start-up message is wrong, and the actual checkpoint LSN matches what we had last written before the server was killed:
In the buffer pool before the server was killed, page 122:0 carries the oldest_modification of 153500937 and FIL_PAGE_LSN of 153500943 (both of which are after the checkpoint). The question should be: Why was fil_space_set_recv_size_and_flags(id=122) not invoked? So, let us look at the recovery once again:
The entire mini-transaction is a single WRITE record for page 122:0, at byte offset 0x39, writing the byte 0x37. Updates of FSP_SPACE_FLAGS are supposed to always be written in full, that is, the start offset must not be more than FSP_HEADER_OFFSET+FSP_SPACE_FLAGS=0x36, and the offset+length must not be less than 0x3a. The problem is unrelated to We appear to have several writes of FSP_SPACE_FLAGS that are missing the mtr_t::FORCED flag to ensure full logging. I will produce a fix shortly. | |||||||||||||||||||||||||||||||||||
| Comment by Marko Mäkelä [ 2020-10-20 ] | |||||||||||||||||||||||||||||||||||
|
The problem appears to be that the |