at first impression I'd make each server having its own trx numbering with no holes.

GTID sequences are per-domain, not per-server. If you want each server to have its own independent GTID sequences, then you can set gtid_domain_id to a different value on each server.

I don't care if it's per domain or per server (although in my opinion the GTID sequence per server is more elegant), the problem here is the consistency, as you can see the number is repeated also in the same domain:

1-100-1000
1-100-1001
1-100-1002
1-200-1003*
1-200-1004*
1-100-1003*
1-100-1004*

When the GTID is generated on the Slave it continues from the next value, when the GTID is again generated on the Master, not having a feedback mechanism, the Master continues where it left and that means that there will be duplicate sequence numbers.

This KB spends some words about it: https://mariadb.com/kb/en/library/gtid/
But it simply suggests not to write on the Slave.

To have a consistent behaviour one should either introduce a feedback mechanism so that the Master can read the latest sequence number from the Slaves, or, the much more easier use a dedicated counter per server.
Anyway this is only an "aesthetical" aspect I suppose.

I don't care if it's per domain or per server (although in my opinion the GTID sequence per server is more elegant), the problem here is the consistency, as you can see the number is repeated also in the same domain:

As far as I understand it, it's not generally a good idea to write to the same domain on two different servers concurrently because of this exact problem. If two servers are writing concurrently, then I believe that it is usually appropriate to use different gtid_domain_id values on each server.

To have a consistent behaviour one should either introduce a feedback mechanism so that the Master can read the latest sequence number from the Slaves, or, the much more easier use a dedicated counter per server.

I believe that setting up circular replication would be the "feedback mechanism" you're looking for, but I think you'd still have to watch out for duplicates that could happen due to slave lag. In general, I think the best option is probably to set gtid_domain_id to different values on servers that will be writing concurrently.

As far as I understand it, it's not generally a good idea to write to the same domain on two different servers concurrently because of this exact problem. If two servers are writing concurrently, then I believe that it is usually appropriate to use different gtid_domain_id values on each server.

To me is too fuzzy. Is the sequence number unique per domain id?
Right now the answer seems to be "I wish"
Not being able to write on a Slave a local insignificant transaction because it messes up the whole GTID doesn't give me the idea of robustness.
If using different domain id is a must in case you may apply a transaction on a Slave I'm fine with it, I'll look for that statement.

To have a consistent behaviour one should either introduce a feedback mechanism so that the Master can read the latest sequence number from the Slaves, or, the much more easier use a dedicated counter per server.
I believe that setting up circular replication would be the "feedback mechanism" you're looking for, but I think you'd still have to watch out for duplicates that could happen due to slave lag. In general, I think the best option is probably to set gtid_domain_id to different values on servers that will be writing concurrently.

Not really. What I have in mind is just a call to the Slaves to ask latest locally generated gtid, of course not really applicable.

To be really global GTID should use a central GTID dispatcher, that whenever a server needs to generate a transaction has a really unique GTID,
but that would be a central 'semaphore' for a whole cluster and the performance impact would make it probably unusable.

Back to the main bug in this report it seems quite bad that STOP/START SLAVE will break replication.

The explication is in how gtid_current_pos is set:

"For each replication domain, if the server ID of the corresponding GTID in @@gtid_binlog_pos is equal to the servers own server_id, and the sequence number is higher than the corresponding GTID in @@gtid_slave_pos, then the GTID from @@gtid_binlog_pos will be used. Otherwise the GTID from @@gtid_slave_pos will be used for that domain."

https://mariadb.com/kb/en/library/gtid/#gtid_current_pos

If I understand correctly, the reported problem is the behavior explicitly documented for current_pos:

Using master_use_gtid=current_pos is probably easiest, as there is then no need to consider whether a server was a master or a slave prior to using CHANGE MASTER. But care must be taken not to inject extra transactions into the binlog on the slave server that are not intended to be replicated to other servers. If such an extra transaction is the most recent when the slave starts, it will be used as the starting point of replication. This will probably fail because that transaction is not present on the master. To avoid local changes on a slave server to go into the binlog, set @@sql_log_bin to 0.

https://mariadb.com/kb/en/library/gtid/
claudio.nanni is it indeed what you are reporting?

Hello Elena,
Yes I reported what is documented, I must have missed that paragraph.
Not sure if it would be the case to point the variable page to that paragraph too:
https://mariadb.com/kb/en/library/gtid/#gtid_current_pos
Or maybe update the documentation of this variable to clearly state that.

As long as it's about documentation, I leave it to documentation experts to decide how to do it best.

Updated the GTID page, added new section from existing text covering current_pos and slave_pos to better emphasize issue.

https://mariadb.com/kb/en/library/gtid/

The slave's I/O thread currently only checks for inconsistent GTIDs when it initializes its local value of gtid_current_pos, which happens when the slave threads are first started. I don't think this is ideal.

If a slave has MASTER_USE_GTID=current_pos set, then I think the slave's I/O thread should periodically compare its local value of gtid_current_pos to the slave's global value of gtid_binlog_pos. This would allow the slave to warn users that its position has become inconsistent, even if the slave threads don't get restarted.

See MDEV-20122 for more information.