HI Benjamin Setzer!

can you provide
my.cnf, show global variables, show global status on both side; plus mysqlbinlog output for some range with Gtid 0-23278185-3223146 in it.

I added the confdir as zip, because we have some customfiles there.
the show global x was done with mysql svs "SHOW GLOBAL variables" > $.txt
Which binlog do you need? master/slave? relaylog from slave?
And how much? It contains real user data, the more you need the more I have to look that nothing sensitive leeks. Also, the original log is gone, but I can reproduce this error in dev.

I tried this in dev with binlog_format = row, then the error disappears and all works as expected.

Hi Benjamin Setzer

Actually master/slave_status/variables file which you provided is wrong , you can do something like this

echo "show global status" | mysql  > file_name

>>Which binlog do you need? master/slave? relaylog from slave?
master and slave binary log (slave relay log is not needed)
>>And how much?

Couple of hundred gtid before the error
Run

MariaDB [(none)]> show master status;

+-------------------+----------+--------------+------------------+

| File              | Position | Binlog_Do_DB | Binlog_Ignore_DB |

+-------------------+----------+--------------+------------------+

| master-bin.000002 |     19208 |              |                  |

+-------------------+----------+--------------+------------------+

use this file name(master-bin.000002)

Show binlog events in binary_log_file_name

If it contains too much log you can simple select last few hundred line using cursor , and if log is less then you can decrease the master-bin.XXXXX no and run show binlog events.

Since it is dev machine and you are able to reproduce error , is it possible to get gdb access to your machine ?

I fixed the status/variables files, I forgot the -e switch.
As for access, I have to ask.

Benjamin Setzer, sachin.setiya.007 Let me jump in. For offline analysis we need mysqlbinlog ouput
of events preceding one that failed with the dup key error:
mysqlbinlog --start-position = pos_start --stop-position= pos_stop binlog.file
here pos_start,pos_stop are offsets in the binlog file. It was suggested pos_start
to be the offset corresponding to some -100th gtid (transaction) prior to the failing one,
and pos_stop value should be of the offset of a gtid that follows the failing one.
You can find them as the number attached to '^# at ' as in an example:

# at 469

#181008 15:13:24 server id 1  end_log_pos 511 CRC32 0x885b83c5  GTID 0-1-2

Alternatively feel free to upload your binlog, I hope it can be made small enough to handle as you wrote the case is reproducible.

cleaning up the binlog is more work than I anticipated.
So I propably won't have time cleaning up some 100 gtids.

There are talks about giving one of you access. You are employed at Maria, right?
This would involve a NonDisclosureAgreement.
This will propably take a while.

Yes, we're the Maria engineering stuff members. Gaining access should definitely simplify resolution as well as If you will be able to reproduce it with relatively small binlog.

Thank you, Benjamin. Andrei Elkin @ MariaDB.com

According to sujatha.sivakumar's search the issue may be caused because of https://bugs.mysql.com/bug.php?id=88321, which is going to be available in a new upstream release.

Hello Benjamin Setzer,

Upstream fix for https://bugs.mysql.com/bug.php?id=88321, is available on
MariaDB 10.3.17 onwards .

commit 7153e155424de4b42bc0f96ceb6433e4ed1f827a
Author: Marko Mäkelä <marko.makela@mariadb.com>
Date: Tue Jul 23 11:01:44 2019 +0300
Revert "MDEV-8827 Duplicate key with auto increment"
This reverts commit ef47b62551b0f37770e5d174ea028150c5b71fd8.
The parent commit 07ba5560da805c766da68811692272dd9eebdaf7
which is a backport of
mysql/mysql-server@1198267c331b045b9cad26be72b1a5b4f8930a79
fixes the issue differently.

Can you please verify the reported bug scenario on latest 10.3 code.

Thank you.

Maybe the upstream bug fixes have dealt with the case, yet that case described as having auto_increment_increment > 1 which seems not to be the current case.

I no longer work at this place, so sadly I cant test it anymore, sorry.

Benjamin Setzer, thank you for the efforts. If we won't complete it wo/ help, you got to file few more!
Sure, otherwise just report them.

Closed in agreement with Sujatha. If a bug which this symptom still exist will proceed from that ticket.