Unfortunately, most InnoDB recovery bugs that we have reproduced and fixed internally have been on the "write" side, that is, there is no way to recover the dataset even after the bug fix is applied.

In 10.11.10 there was also the fix MDEV-34830, which makes recovery catch more cases of inconsistency and to refuse normal recovery without innodb_force_recovery.

I’m afraid that there is not much that we can do about this corruption. If it has already been fixed in MDEV-34689, then you would need to restore MariaDB Server 10.11.10 from a logical dump and see if this corruption stays away. If the InnoDB files were corrupted already by 10.11.9, there would be no way to magically fix it, other than restoring the database from a logical dump.

You claim that MariaDB Server 10.11.10 is affected by this bug. Can you clarify if this is repeatable on a copy of database files that recovered fine with 10.11.9?

If by "nodes" you are referring to a Galera cluster and you are using the default setting wsrep_sst_method=rsync, you need to be aware of MDEV-32115, which is still open.

Thanks for the detailed explanation and the heads-up, Marko.

Looking back at that one case, where an upgrade saved the dataset after it failed, it seems https://jira.mariadb.org/browse/MDEV-34830 covered it. The error wasn't exactly `Missing FILE_CHECKPOINT` , my apologies.

[49148]: 2024-11-04  2:18:31 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=99252666420468

[49148]: 2024-11-04  2:18:31 0 [Note] InnoDB: At LSN: 99252785562429: unable to open file ./u8329

[49148]: 2024-11-04  2:18:31 0 [Warning] InnoDB: Ignoring corrupted page identifier at LSN 99252840906101

[49148]: 2024-11-04  2:18:31 0 [Warning] InnoDB: Ignoring malformed log record at LSN 99252840906101

[49148]: 2024-11-04  2:18:31 0 [Warning] InnoDB: Ignoring malformed log record at LSN 99252840906101

[49148]: 2024-11-04  2:18:31 0 [Warning] InnoDB: Ignoring corrupted page identifier at LSN 99252840906101

[49148]: 2024-11-04  2:18:31 0 [Warning] InnoDB: Ignoring corrupted page identifier at LSN 99252840906101

[49148]: 2024-11-04  2:18:31 0 [Warning] InnoDB: Ignoring corrupted page identifier at LSN 99252840906101

[49148]: 2024-11-04  2:18:31 0 [Warning] InnoDB: Ignoring malformed log record at LSN 99252840906101

[49148]: 2024-11-04  2:18:31 0 [Warning] InnoDB: Ignoring malformed log record at LSN 99252840906101

[49148]: 2024-11-04  2:18:31 0 [Warning] InnoDB: Ignoring corrupted page identifier at LSN 99252840906101

[49148]: 2024-11-04  2:18:31 0 [Warning] InnoDB: Ignoring corrupted page identifier at LSN 99252840906101

[49148]: 2024-11-04  2:18:31 0 [Note] InnoDB: End of log at LSN=99252844533207

I guess this one success prompted us to think that maybe it was meant to handle `Missing FILE_CHECKPOINT` as well, alas.

The post concerned recovery after it happened on 10.11.9 and whether it could be restored to a working state without dumping and reimporting hundreds of GBs of data. I'm keeping my fingers crossed that corruption will no longer occur on the latest version.

arnklo, that error looks more like something about the log record itself and should be a separate bug on its own.

Was the file name ./u8329 really output like that, or did you anonymize the log? In this log file format, there is a checksum after each mini-transaction. The end of the log is 3,627,106 bytes after the problematic LSN 99252840906101. The maximum size of a mini-transaction is less than that, 2 megabytes. So, the log definitely was successfully parsed after that point. I think that this is a new type of an error, and I would pretty much want to have a copy of the relevant portion of the log. To start with, the log file header should indicate the mapping between LSN and file offsets:

od -Ax -t x1 -N 12288 ib_logfile0

This part does not include any application data; the log records start right after the offset 12288. I would be interested in a few megabytes of log right before the end; I’d give instructions for dumping those based on the output of the above. That you’d of course share outside this Jira ticket, if you are willing to share it.

Do you use innodb_encrypt_log=ON, per chance?

Hey Marko,

I'm sorry I couldn't reply earlier.

That log was indeed anonymized. `./

/#sql-ib11573128.ibd for tablespace 11573116`

Here's the output from the command from the server that managed to recover.

$ od -Ax -t x1 -N 12288 ib_logfile0

000000 50 68 79 73 00 00 00 00 00 00 4b 7c 76 8e 97 dd

000010 4d 61 72 69 61 44 42 20 31 30 2e 31 31 2e 36 00

000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

*

0001f0 00 00 00 00 00 00 00 00 00 00 00 00 ee 1d d2 59

000200 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

*

001000 00 00 5b 01 ff b5 1f 04 00 00 5b 02 0b 39 f3 03

001010 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

*

001030 00 00 00 00 00 00 00 00 00 00 00 00 ed c0 e4 99

001040 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

*

002000 00 00 5b 01 ff 99 ba 1d 00 00 5b 02 0b 2a 9b 29

002010 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

*

002030 00 00 00 00 00 00 00 00 00 00 00 00 48 c3 f2 60

002040 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

*

003000

And here's from one that didn't:

$ od -Ax -t x1 -N 12288 ib_logfile0

000000 50 68 79 73 00 00 00 00 00 00 00 00 00 00 b6 fe

000010 4d 61 72 69 61 44 42 20 31 30 2e 31 31 2e 37 00

000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

*

0001f0 00 00 00 00 00 00 00 00 00 00 00 00 9e 1c 84 7b

000200 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

*

001000 00 00 3b 1c d6 db 01 18 00 00 3b 1d 3a e4 93 1b

001010 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

*

001030 00 00 00 00 00 00 00 00 00 00 00 00 a8 67 f4 7b

001040 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

*

002000 00 00 3b 1c e3 34 2d d4 00 00 3b 1d 3d df 26 5c

002010 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

*

002030 00 00 00 00 00 00 00 00 00 00 00 00 57 e2 d2 4c

002040 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

*

003000

No, we do not use innodb_encrypt_log. I hope this helps.

In the dump of an ib_logfile0 that could not be recovered, at file offset 8 we have the 64-bit big endian integer 0xb6fe (46846), which indicates that the log file offset 0x3000 corresponds to the LSN 0xb6fe. The log corruption was reported near the LSN 99252840906101, which would be at file offset 99252840940659 or 0x5a451a53c473. Is the corrupted log file really that large (about 90 TiB), or was this header copied from another log file? Also, the log checkpoint LSN would be about 0x1f0000000000 (31 TiB) behind the recovery position. I find it very unlikely that this dump is from the file for which the InnoDB: Ignoring corrupted page identifier was reported.

Hey Marko,

Interesting findings. "Ignoring corrupted page identifier" was only on that single server that recovered; it's the first header. The non-recovered ones halted on "Missing FILE_CHECKPOINT. " The second header is from one of those servers, so this makes sense. The ib_logfile0 for all servers is 2GB large. I hope that clears things up a bit.

Hi arnklo, let me try again with the first od output. 99252840906101-0x4b7c768e97dd+0x3000 would correspond to the file offset 16,254,903,834,008 or 15,138 GiB, which is much larger than 2 GiB. But I forgot that the log file would wrap around from the end to 0x3000. So, we have to take the modulus of 16254903834008%(0x80000000-0x3000) = 693,110,168. If this were the right file, I would like to see a few hundred bytes of the log file contents around this offset.

Your log output displayed the checkpoint LSN 99252666420468 (0x5a450fecccf4). The largest of the 64-bit big endian checkpoint LSNs at 0x1000 and 0x2000 in the first dump is 0x5b01ffb51f04, which is somewhat larger. So, this might not be the right file. In the second od output, the checkpoint LSN is much smaller, 0x3b1ce3342dd4.

Do you still have a copy of the ib_logfile0 that is associated with the log parsing error around LSN 99252840906101?