Update: after the last replication outage (2018-11-14) we added more slaves like described above. So we had the following machines:

• DB02 as Master with MariaDB 10.1
• DB01 as SLAVE with MariaDB 10.1
• DB03 as SLAVE with MariaDB 10.3
• DB04 as SLAVE with downgrade to MySQL 5.6

Additionaly we changed the binlog format to ROW instead of MIXED

at 2018-12-11 we had another outage and this time it's even worse... :

• SHOW SLAVE STATUS on all machines still shows zero replication lag... and all seems OK !
• But for DB01 + DB03 (both mariadb) no more changes happen (added orders on DB02 are not added on DB01 + DB03) -> but Exec_Master_Log_Pos is increasing
• DB04 (MySQL) seems to run as expected (no replication lag and changes on DB02 get applied)
• It seems there is some periodically now in the outages: the second one happend after 18, the third after 19 days...

any ideas?

Thanks for reporting, Maik!

It's difficult to grasp your setup. If I got it right (as you say 'master-master' and 'cluster'), two servers replicated crisscross DB01 <-> DB02 ?
I understood that you suspect that an INSERT (Write_rows_log_event) was not applied by that slave.

If it's really crisscross setup, I naturally wonder how it's guaranteed that local data modification does not conflict with replication.
Assuming it's all proved to be safe for you and then indeed it's replication to blame, you need continue with your method to identify
a missed-to-apply replication event in the master binlog, also check the entire transaction that it belongs to whether it was skipped entirely. Please tell us when such event/transaction will be firmly identified, so we could
check that closer.

Also it would be helpful to provide

show global status,  show global variables, my.cnf of master and slaves

SHOW ALL SLAVES STATUS on the slaves.

Good luck,

Andrei

Hey Andrei,

almost right...
it was a master/master early 2018 and we used the following methods to prevent conflicts:
1. Use Keepalive to provide a ClusterIP (each server in BACKUP Mode so the ip is not "flapping"
2. Allow access to the Database only via the ClusterIP
3. Configure each server with a different another auto_increment_offset (1 and 2) and set auto_increment_increment to 2.

But after the first Replication Problem we moved to a Master/Slave (by get a snapshot of the last master (DB02) and configure it as a slave...) So since 23th October we no longer have a "crisscross" Setup...

i've attached db03.txt which contains the requested outputs of one of the mariadb slave (DB03) which is no longer applying changes. As well as db02.my.conf (MASTER my.cnf) and db03.my.cnf (SLAVE my.cnf)

do you have any idea whats the difference between MariaDB 10.1 and MySQL 5.6 regarding replication and could cause such a behaviour?

> 3. Configure each server with a different another auto_increment_offset (1 and 2) and set auto_increment_increment to 2.

Sure in such case auto-increment INSERT:s are okay, but there are UPDATE:s according to your snippet.
I am not convinced yet in conflict-free master-master of your setup.

However as you clarified the missed record error has occurred now in the master-slave.
Let me understand the very error now.

A piece of your comments start with

> The Replication on DB01 stopped with exec master position at 68765489

to say the error must have happened at applying

mysql-bin.001407 	68765489 	Write_rows_v1 	        12 	    68765660 	table_id: 4297936365 flags: STMT_END_F

but I could not get the meaning of the 2nd part of the comments:

>and complaining the insert at 68767005

which refer to a following event.

I need your explanation.

I could imagine that the error actually happened at the 2nd INSERT of 68767005, and you found at your analysis it's the 1st 68765489 that did not make its job. But I can't read it this way.

> whats the difference between MariaDB 10.1 and MySQL 5.6 regarding replication

Generally we have few differences, but I don't see how any of them can relate to your case.

i think you read it right...
As you say, the second comment (insert at 68767005) stopped the replication with an foreign key constraint violation, which happend because the first one (68765489) was not applied.. (and more worse it raised no error! so some kind of silent ignored...)

>the second comment (insert at 68767005) stopped the replication with an foreign key constraint violation

That means BEGIN GTID 0-12-569302375 transaction (gtid group) which this insert belongs to.
And the suspected insert belongs to a preceding one.
And this looks quite easy to analyze then.

If you have a slave "frozen" after this error I could try to help you to unroll what might have happened.
Otherwise we seem have to wait for perhaps another period that you spotted.

table_id overflow?

FYI：https://jira.mariadb.org/browse/MDEV-18175

2019-01-08 we hit another replication outage...
The table_id idea is interesting, it seems that every time we hit this Problem, table_id is greater than 2^32 - 1 (4297936365 last, 4296337452 current) (https://jira.mariadb.org/browse/MDEV-18175 says something about > 17179869184 which is wired to me because either uint is 32 or 64 bits which both would'nt match that number?). Is the computation of the table_id somehow related to the number of tables? [Andrei Elkin] we currently have such a frozen slave, is some kind of remote session possible for you?

Maik
yes，you are right，it's 2^32 - 1.

ok, for me it sounds really close to our problem...

Last Events wich are inserted at slave:

#190108 7:51:30 server id 12 end_log_pos 62304235 Table_map: `supremeshops`.`supr_send_cloud` mapped to number 4294743012

1. at 62304235
   #190108 7:51:30 server id 12 end_log_pos 62304406 Write_rows: table id 4289245655 flags: STMT_END_F

And the Problem begins with:

1. at 71616759
   #190108 8:03:42 server id 12 end_log_pos 71616827 Table_map: `supremeshops`.`supr_send_cloud` mapped to number 4296337457
2. at 71616827
   #190108 8:03:42 server id 12 end_log_pos 71616998 Write_rows: table id 4296337452 flags: STMT_END_F

From that Point we have a table_id > 2^32 - 1 ...

so if i got it right the Problem is slave related and we could prove that this is our Problem bei starting a fixed slave and attach it to the binlog position with the silent dropped insert. The fixed slave now should replicate again, right?

mapa Your latest data points to MDEV-18175, indeed. I'll try to come up with a patch shortly.

Elkin i've changed the uint table_id in sql/table.h to ulong table_id and reattached the slave like described above -> Works

Elkin do you known if the table_id maybe increasing faster because of we have ~800k tables in one database?

mapa Right, the reason is the short type of TABLE_LIST::table_id. Well done!

table id in TABLE_SHARE::table_map_id changes by each ALTER. And it's of 8 bytes.

just to be sure.... Is there something more to change? https://github.com/mysql/mysql-server/commit/c9c7bda021b8d4d7aa997f24e301df5b76dfe215

maybe it's a good idea to use a typedef for table_id to have the same type across the entire code, like mysql does it?

sachin.setiya.007, there has been activity on two tickets - MDEV-17803, MDEV-18175 with couple of people involved. Since the older one had had some history I could not resist from fixing it. The 2nd is a dup. Could you please check out the patch?

mapa, sorry, we seem to have been working on the same thing without coordination. I appreciate your help!

Elkin that's just me quick and dirty change i've done to test/prove that this is the reason of our Problem... so it is i thought i should contribute it because i did not excpected that you find time to start on that topic that fast . Cause i only have that short qnd fix please push/PR your changes, i close my PR?

mapa We always love to have contributions. ruochen and yourself did a great job, thank you both!
I could not defer (and said that yesterday with 'shortly') with this case fast having seen its history and active interest of few people.

Yes, please close PR, my "internal" patch is being reviewed.

Patch cce2b45c8f5b32 was actually pushed back in January.