Howdy Brandon!

Could you please have a look at bb-10.11-andrei?
You may not be the only reviewer, but let me pick you first.

Cheers,
Andrei

marko, thank you for attending this ticket!

Sure Innodb team needs to look at the patch. Actually it was under way already in form of MDEV-34466 that I had to find out as it was blocking my progress on these fixes (that include a hunk for the latter bug). Btw they do not really change innodb_lock_wait_timeout policy, at least never beyond replication.
To view this work as removing limitations (which they are) of engine/server locking protocol started by MDEV-26682 and followed in
MDEV-33454 is much more fair in my opinion.

As to the comparison of this and Kristian's method of resolving MDEV-32020, I only can repeat for what's been said multiple times.
Arguably they are not mutually exclusive.
Yet MDEV-742 is clearly a preferred choice when the user requires failover to slave.
As it has to be reliable and fast the method of collecting and deferred applying of XA events simply may not be an option ('cos
it *hopes* on replaying would succeed while time to spend on that is always affordable).
Neither I am certain that implementation of collecting for deferred applying XA events is really straightforward.

> Neither I am certain that implementation of collecting for deferred applying XA events is really straightforward.

There is already a (prototype) implementation of this, see knielsen_mdev32020

> Yet MDEV-742 is clearly a preferred choice when the user requires failover to slave.

Really? The failing over of a prepared XA transaction is a (very) rare operation. The normal apply of an XA transaction will occur thousands or millions of times more often. Applying the XA PREPARE on the slave pessimises the common operation by doubling the work on the slave to process two event groups, two GTIDs, and two commits inside InnoDB.

I think there's a misconception that somehow the XA PREPAREd transaction will normally be already applied on the slave in the case where a failover occurs. That's unlikely to be the case, especially for transactions that takes longer to apply, as the commit on the master will normally arrive shortly after the prepare, while the slave can only start applying the xa prepare after it has been synced to the binlog on the master.

For the rare user that really wants to recover an XA PREPAREd (but not committed) master transaction on the slave, it will be necessary to apply the to-be-recovered prepare on the slave. The requirements for this are similar, though it is simplified in the MDEV-32020 proposal since there is no requirement to apply them in order like in current code. However, there is no need to do so for the wast majority of transactions that are prepared+committed on the master, that just introduces a lot of unnecessary overhead and complications.

>> Yet MDEV-742 is clearly a preferred choice when the user requires failover to slave.

>Really? The failing over of a prepared XA transaction is a (very) rare operation.

Sorry, 'rare operation' is something none, with all due respect to you dear Kristian, but an actual user can claim.
And we don't know the future either.
Similarly to 'normally' here
> as the commit on the master will normally arrive shortly after the prepare
Let's admit that it's all your adjectives and assumptions that may not come true.
To the usability matter, also please refer to mysql@oracle state of xa replication.
I've not heard from them, having this solution in place since 2013, any ideas to cancel it.

I suggest to leave this matter alone. It's a firm and obvious fact that only the eager replication of
XA-prepare provides instant recovery.

This approach can take its toll Actually not, find here "extra fsync" dismissal.
> two GTIDs, and two commits inside InnoDB.
However it's not about 'doubling the work' at all. I and Brandon shared MDEV-31949 consoling benchmarkings. There's no reason to doubt they could be improved still.

Howdy Susil!

Could you please concurrency tests to prove the fixes.
They are currently pushed to bb-10.6-andrei to base on a not-fully completed MDEV-34466 branch. That was necessary to avoid some errors of the engine in a service that XA replication relies on.
Read more on what the patch is about in the commit message as well.
An mtr rpl suite test should be helping for insights how/what to test.
In brief I consider an arbitrary size worker pool, multiple clients on master running mixed
normal and XA transactions updating non-unique-only (including NULL-able unique) index
tables in ROW format.
All the sequential and the parallel optimistic or conservative slaves should complete the work
with consistent data and gtid state in the end.

GTID-connection mode (aka Change-Master-to master_use_gtdi) is irrelevant.
Gtid strict mode should be on.

Some comments on the patch: https://lists.mariadb.org/hyperkitty/list/developers@lists.mariadb.org/thread/KFTWZ2CCNRFDJ77B4G4TGIHXMVMCFVHC/

I suggest that this be tested also with innodb_snapshot_isolation=ON. Hopefully it will replace some lock waits with other errors (ER_LOCK_WAIT_TIMEOUT, ER_CHECKREAD).