A Tool to Compare Replication Servers After a Crash

After a primary crashes, a DBA must establish which transactions each
server holds before deciding which one continues as primary. The
procedure today is to read @@gtid_binlog_pos on both servers,
compare sequence numbers per domain by hand, then guess a start position
for mariadb-binlog to see what the difference contains. Two limits
make the result unreliable:

• Comparing @@gtid_binlog_pos cannot detect that the two histories
  have parted. The position holds one domain-server-seq triple per
  domain, describing only the last transaction logged in that domain.
  After a failover where the old primary kept writing, both servers hold
  different transactions carrying the same domain_id and the same
  seq_no, and the two positions still compare cleanly as one being
  ahead of the other.
• The comparison says nothing about whether the differing transactions
  are still available. A count decides nothing on its own, because
  catching a server up is possible only while the events remain in the
  other server's binary log.

The idea of this ticket is to develop a tool (or possibly extend
mariadb-binlog) that compares the binlog state of two servers (or
sets of binary logs). It would output:

• whether the two are identical,
• per domain, which transactions each server holds that the other does
  not,
• whether the two histories have parted, and at which domain and
  sequence number,
• whether those transactions are still present in the binary logs, so
  whether catching a server up is possible at all,
• optionally, the transactions themselves, written out as two binary
  logs, one per direction.

Writing the difference out as binary logs shows the DBA the exact
transactions rather than a count, and each file is then the input to
whichever reconciliation is chosen: applying it to the server that is
behind, or generating the inverse with mariadb-binlog --flashback.

It could look something like this:

$ mariadb-binlog-diff --server-a=... --server-b=... [--extract]

Domain 0:

  (..., 0-1-50) : Missing

  [0-1-50, 0-1-100] : Consistent

  [0-1-101, 0-1-105] : Diverge (5 transaction present in A's binlog, not B's)

  [0-2-101, 0-1-108] : Diverge (8 transaction present in B's binlog, not A's)

Domain 1:

  [1-1-1, 1-1-100] : Consistent

Verdict: the histories have diverged in domain 0 starting after 0-1-100.

With two optional additional sets of binlog files (if --extract is
given):

• trx_in_A_not_B.000001, trx_in_A_not_B.000002, ...
• trx_in_B_not_A.000001, trx_in_B_not_A.000002, ...

Additional considerations for this project that must be addressed in the
design:
1. If a slave is running without binlog enabled, how is its state
reported? Should the server use @@global.gtid_current_pos to
determine state? Note that the binlog will not be available in such
a configuration, this is just for an initial level of stateful
analysis.
2. Should the tool extend mariadb-binlog or be a new CLI binary?

Note this is closely tied with:

• MDEV-40729 (which provides additional logic to identify if a server
  has actually applied a given transaction from a client connection).
• MDEV-37605 (which extends mariadb-binlog with logic to write a
  binlog file, where it is coverted from an innodb-binlog file).