The assertion fails because a corruption of the doubly linked list of sibling pages is noticed. Apparently the same error is reported in a friendlier way by CHECK TABLE.

We’d better propagate the error to the client connection, instead of crashing the server. That is the topic of MDEV-13542.

Normally, such inconsistency between linked pages is prevented by atomic mini-transactions, which combine page locking and logging. Any operation that would modify multiple pages is logged as an atomic mini-transaction. If for some reason the ib_logfile0 is not being copied in a consistent way and some logged changes are lost, you can get such corruption. (Note: Also starting up the server with innodb_force_recovery=6 or without the ib_logfile0 file is a recipe for disaster.)

Because Galera has been mentioned, I would primarily suspect an error in the snapshot transfer mechanisms that were used. Any "backup" or "restore" step could potentially cause this type of corruption. Can you trace the history of this data directory in terms of such corruption?

Hi, I work with Dan and was involved with this incident. This is a really interesting theory! I guess I had just assumed that the corruption happened and then the state transfer spread it to the rest of the cluster. It hadn't occurred to me that a state transfer could be the source of the corruption.

We can't precisely trace the origin of this data directory, but I can tell you a couple of things about it. This data directory has been passed around from node to node, because each of our nodes is wiped and re-installed from the ground up every two weeks. We have always used mariabackup-based state transfers. Over the life of this database, no node has been restored from an external backup and joined the cluster. It is easily possible that corruption was introduced during an SST and then passed to the rest of the nodes through subsequent SSTs.

Based on your theory, we're wondering if we should switch to the mysqldump or rsync SST method. We care quite a bit about the integrity of the data in this cluster, and much less about the speed of SSTs. Do you feel that mariabackup is risky? Could the data itself be at risk and not just the indices?

And if a mariabackup state transfer is risky, maybe Galera should run a CHECK TABLES after the SST completes just to be safe?

I am not deeply familiar with the Galera cluster or its snapshot transfers (SST), but I know InnoDB.

Theoretically speaking, the mysqldump method should be safest. But it would also likely be the slowest.

The rsync method is inherently risky if the source can theoretically be modified during the transfer. I suppose that this method would be protected by the setting innodb_disallow_writes, which causes some problems on its own (MDEV-24845). I have not checked if all relevant writes are being blocked by that setting. (There are also writes outside InnoDB.)

The mariabackup a.k.a. mariadb-backup method is what we would recommend for backups. It is possible that you are hitting some bug in it. It is also possible that the Galera SST scripting around it is ignoring some error, and letting corruption propagate. A repeatable test case that mimics your workload (similar table structures and DML operations) would be very much appreciated.

Thanks, that agrees with my impression that mysqldump is the safest. We have been using mariabackup for backups as well as SSTs. One of our learnings from this incident is that we also need to do regular mysqldump backups, because this corruption was propagated into our backups.

A repeatable test case is not feasible, I'm afraid. The exact circumstances that introduced the corruption are unknown, and we also have no clear idea when or how the corruption was introduced. The only thing we do know is that our query pattern did not change, nor did our cluster management / SST patterns.

In MDEV-24845 I wrote some notes how I suspect that the rsync snapshot method is currently broken except when innodb_use_native_aio=0 is being used.

I believe that with a correct implementation of necessary and sufficient locks, the rsync snapshot method should be the safest and fastest. But, currently we do not seem to be there.

The error message about corrupted FIL_PAGE_NEXT or FIL_PAGE_PREV links is written to the error log by CHECK TABLE.

The assertion that failed and caused the server crash is in the function btr_cur_latch_leaves():

                        ut_a(btr_page_get_prev(get_block->frame)

                             == block->page.id().page_no());

I have rewritten that function in MDEV-13542 which is currently being tested. This particular assertion has been removed; it is not being tested even in the rewritten function btr_page_get(). I think that it is acceptable, because the purpose of btr_cur_latch_leaves() is to only acquire latches on the sibling pages if those pages can be loaded. The caller would check for errors elsewhere, when actually accessing those sibling pages. I realize that I still have some work to do, to ensure that such conditions are in place and to replace similar assertions in btr_discard_page() and btr_attach_half_pages() and possibly elsewhere.

When it comes to the root cause of this corruption, I believe that the fix of MDEV-25975 made the default wsrep_sst_method=rsync actually safe to use. I think that this ticket is about corruption caused by Galera snapshot transfer, while MDEV-13542 is about preventing crashes due to corruption.

If wsrep_sst_method=mariabackup was used, this corruption might be explained by MDEV-28731. The crash on this corruption was fixed by MDEV-13542.