Thanks for testing this!

You are in uncharted territory, I did consider circular topologies in the
design but did not test yet

There is one problem with your test. You have two masters active at the same
time. Doing this with GTID requires configuring different gtid_domain_id for
the two masters.

It does not help that you stopped the direction 2->1. What matters is that you
have two masters (whether their slave is running or not at the precise
moment), and you are doing updates on one without first replicating all
changes from the other.

Concretely, we have

S2: create table t2 ...
S1: insert into t1 ...

On S2, "create table t2" will be binlogged before "insert into t1". But on S1,
"insert into t1" is binlogged first.

So when S1 connects with GTID as slave to S2, it asks to start from the
"insert into t1" which is the latest GTID it applied in domain 0. But this is
after "create table t2" in binlog on S2, so that event is lost.

There are two ways to do this correctly:

1. Either configure different domain_ids for S1 and S2.

2. Or alternatively, use a single domain and make sure that everything is
replicated S2->S1 before doing changes on S1, and vice versa. Basically, let
both slave directions run at the same time, and --sync-with-master each time
before doing updates on the next server.

So I would like to handle this in two stages.

First, we should make sure that (1) and (2) actually work correctly (they
should, but it definitely needs testing, there are likely bugs).

Second, while what the test does is fundamentally incorrect and cannot work,
it would still be best if we can give a better user experience, give a clear
error message rather than silently dropping events.

"First" we should do now. "Second" I would like to revisit later, when the
basic stuff has been better tested and is solid. Multimaster ring is somewhat
of an advanced concept, and it is more reasonable to expect more knowledge
from the DBA who sets that up. My vague idea is that we could implement a GTID
"strict" mode that would detect the wrong configuration and give an error
immediately. The detection would be to see that S2 gets an event from S1 with
the same sequence number that it already logged itself to its own binlog. Such
strict mode can probably not be on by default though, as then a simple upgrade
to 10.0 would break ring setups, even if users have no plans to use GTID.

What do you think?

BTW, I've now started writing some docs!

I didn't manage to finish them today, but should be ready early next week, they should explain this issue, among others.

>> There is one problem with your test.

I see, thanks for the explanation.
I have a feeling it's going to be a tough sell, on two main reasons:

reason A:
the holy cow of anything new: "it used to work before" (in this case, with the old-style replication). While I know that it begs the question "so what?", I also know that every single user who runs onto a problem will give it as an argument, and many, being unhappy about the explanation, will spread the word that the new replication is unsafe and causes regressions.

reason B:
logically, if you forget the implementation, there is nothing in the flow that can possibly create a conflict. With data streams being truly independent one can reasonably expect that they should not affect each other .
From real-life perspective, I think the solution 2 ("make sure that everything is replicated S2->S1 before doing changes on S1") is inapplicable. Forget the circular replication, it's just to make MTR do some preliminary setup for me.
Lets say you have 1->2 replication (or possibly 1->2->3 replication).
Pretty much every scenario that comes from a user and somehow involves replication shows that the slave is never totally read-only. Something always happens on slave – maintenance work, some statistics collection, analytics, whatever. A part of it will inevitably hit the binary log. Even FLUSH TABLES goes to the binlog! And realistically, it's impossible to shield the binary log from all this stuff.
So you have all that and don't expect any problem, but suddenly you need to promote server 2 as a master, and make server 1 its slave. And it turns out that you have a few events in the server 2 log which are totally innocent, but they are intervened with server 1 events and hence will cause the problem.
The solution 1 (different domain IDs) is possible indeed. I just start having a hard time finding a scenario when domain IDs can actually stay the same on different servers with any reasonable safety... I mean, it turns out that even if you don't expect your master and slave to switch places, ever, you still have to set different domain IDs, just in case, because you can't be sure what will happen in the future, right?

>> So I would like to handle this in two stages.
Okay.

Decreasing the priority since it works by design, leaving open for the second stage described above; not removing 10.0.3 from the 'Fix version' list, since for minor bugs it gets updated automatically upon a release anyway.

I had a good discussion on this with Elena.

I think the root problem here is that when we do "insert into t1 values (2)" on server 1, this changes the replication position of server 1, so that it now skips earlier transactions on server 2. This is highly unexpected.

The original reason for this was to make CHANGE MASTER TO master_use_gtid=1 work automatically, regardless of whether the server was before the CHANGE MASTER already a slave server, or in fact the old master of the replication hierarchy.

However, now I think this magic causes more problems than it solves. I think I have an idea for doing this better, in a way that avoids the surprising magic but still makes it easy to connect a server with GTID, whether it was a master before or a slave.

After implementing CHANGE MASTER TO master_use_gtid=slave_pos (and using it in the test case instead of master_use_gtid=1), the test passes without problems. So this is promising.

This is now fixed and pushed to 10.0-base.

The fix is the new user interface with separate master_use_gtid=slave_pos and
master_use_gtid=current_pos. With slave_pos, the test case passes without
problems.